Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]A Chat only can answer one question when using chatqna-conversation-ui:1.0 #795

Open
2 of 6 tasks
hualongfeng opened this issue Feb 11, 2025 · 1 comment
Open
2 of 6 tasks

[Bug]A Chat only can answer one question when using chatqna-conversation-ui:1.0 #795

hualongfeng opened this issue Feb 11, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@hualongfeng
Copy link

hualongfeng commented Feb 11, 2025 •
edited
Loading

Priority

P3-Medium

OS type

Ubuntu

Hardware type

GPU-Nvidia

Installation method

  • Pull docker images from hub.docker.com
  • Build docker images from source
  • Other

Deploy method

  • Kubernetes Helm Charts
  • Kubernetes GMC
  • Other

Running nodes

Single Node

What's the version?

opea 1.0

Description

A Chat only can answer one question. If I answer second question, it still response the first question.

Reproduce steps

values-deepseek.yaml

# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

# Default values for chatqna.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: opea/chatqna
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "1.0"

port: 8888
service:
  type: ClusterIP
  port: 8888

securityContext:
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  runAsNonRoot: true
  runAsUser: 1000
  capabilities:
    drop:
    - ALL
  seccompProfile:
    type: RuntimeDefault

nodeSelector: {}

tolerations: []

affinity: {}

# This is just to avoid Helm errors when HPA is NOT used
# (use hpa-values.yaml files to actually enable HPA).
horizontalPodAutoscaler:
  enabled: false

# Override values in specific subcharts
tgi:
  LLM_MODEL_ID: deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
  accelDevice: "nvidia"
  image:
    repository: ghcr.io/huggingface/text-generation-inference
    tag: "2.2.0"
  resources:
    limits:
      nvidia.com/gpu: 4
  livenessProbe:
    initialDelaySeconds: 5
    periodSeconds: 5
    timeoutSeconds: 1
  readinessProbe:
    initialDelaySeconds: 5
    periodSeconds: 5
    timeoutSeconds: 1
  startupProbe:
    initialDelaySeconds: 5
    periodSeconds: 5
    timeoutSeconds: 1
    failureThreshold: 12000

# disable guardrails-usvc by default
# See guardrails-values.yaml for guardrail related options
guardrails-usvc:
  enabled: false

global:
  http_proxy: ""
  https_proxy: "http://proxy.ims.intel.com:911"
  no_proxy: "chatqna-chatqna-ui,chatqna-ui,chatqna-data-prep,data-prep,chatqna-embedding-usvc,embedding-usvc,embedding-svc,chatqna-llm-uservice,llm-uservice,llm-svc,chatqna-redis-vector-db,redis-vector-db,chatqna-reranking-usvc,reranking-usvc,reranking-svc,chatqna-retriever-usvc,retriever-usvc,retriever-svc,chatqna-tei,tei,chatqna-teirerank,teirerank,chatqna-tgi,tgi,chatqna-nginx,chatqna,chatqna-chatqna-ui,chatqna-ui,192.168.0.0/24,127.0.0.1,localhost,.intel.com,.default.svc.cluster.local,10.96.0.0/12,10.244.0.0/16"
  HUGGINGFACEHUB_API_TOKEN: "insert you token"
  huggingfacehub_api_token: "insert you token"
  # HF_ENDPOINT: "https://hf-mirror.com"
  # set modelUseHostPath or modelUsePVC to use model cache.
  # modelUseHostPath: ""
  modelUseHostPath: /mnt/s3-mount

  # Prometheus Helm installation info for subchart serviceMonitors
  prometheusRelease: prometheus-stack

Run chatqna:
helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=/mnt/s3-mount/ -f chatqna/values-deepseek.yaml

Raw log

2025-02-11T05:49:02.754064Z  INFO text_generation_launcher: Args {
    model_id: "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
    revision: None,
    validation_workers: 2,
    sharded: None,
    num_shard: None,
    quantize: None,
    speculate: None,
    dtype: None,
    trust_remote_code: false,
    max_concurrent_requests: 128,
    max_best_of: 2,
    max_stop_sequences: 4,
    max_top_n_tokens: 5,
    max_input_tokens: None,
    max_input_length: None,
    max_total_tokens: None,
    waiting_served_ratio: 0.3,
    max_batch_prefill_tokens: None,
    max_batch_total_tokens: None,
    max_waiting_tokens: 20,
    max_batch_size: None,
    cuda_graphs: Some(
        [
            0,
        ],
    ),
    hostname: "chatqna-tgi-76c78f4c54-r4scd",
    port: 2080,
    shard_uds_path: "/tmp/text-generation-server",
    master_addr: "localhost",
    master_port: 29500,
    huggingface_hub_cache: Some(
        "/data",
    ),
    weights_cache_override: None,
    disable_custom_kernels: false,
    cuda_memory_fraction: 1.0,
    rope_scaling: None,
    rope_factor: None,
    json_output: false,
    otlp_endpoint: None,
    otlp_service_name: "text-generation-inference.router",
    cors_allow_origin: [],
    watermark_gamma: None,
    watermark_delta: None,
    ngrok: false,
    ngrok_authtoken: None,
    ngrok_edge: None,
    tokenizer_config_path: None,
    disable_grammar_support: false,
    env: false,
    max_client_batch_size: 4,
    lora_adapters: None,
    disable_usage_stats: false,
    disable_crash_reports: false,
}
2025-02-11T05:49:02.754174Z  INFO hf_hub: Token file not found "/tmp/.cache/huggingface/token"    
2025-02-11T05:49:07.509615Z  INFO text_generation_launcher: Model supports up to 131072 but tgi will now set its default to 4096 instead. This is to save VRAM by refusing large prompts in order to allow more users on the same hardware. You can increase that size using `--max-batch-prefill-tokens=131122 --max-total-tokens=131072 --max-input-tokens=131071`.
2025-02-11T05:49:07.509648Z  INFO text_generation_launcher: Default `max_input_tokens` to 4095
2025-02-11T05:49:07.509655Z  INFO text_generation_launcher: Default `max_total_tokens` to 4096
2025-02-11T05:49:07.509659Z  INFO text_generation_launcher: Default `max_batch_prefill_tokens` to 4145
2025-02-11T05:49:07.509690Z  INFO text_generation_launcher: Sharding model on 4 processes
2025-02-11T05:49:07.509994Z  INFO download: text_generation_launcher: Starting check and download process for deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
2025-02-11T05:49:13.286145Z  INFO text_generation_launcher: Files are already present on the host. Skipping download.
2025-02-11T05:49:14.016397Z  INFO download: text_generation_launcher: Successfully downloaded weights for deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
2025-02-11T05:49:14.016906Z  INFO shard-manager: text_generation_launcher: Starting shard rank=0
2025-02-11T05:49:14.016983Z  INFO shard-manager: text_generation_launcher: Starting shard rank=1
2025-02-11T05:49:14.017140Z  INFO shard-manager: text_generation_launcher: Starting shard rank=2
2025-02-11T05:49:14.017250Z  INFO shard-manager: text_generation_launcher: Starting shard rank=3
2025-02-11T05:49:24.026857Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:49:24.027027Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:49:24.027355Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:49:24.027630Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:49:34.033869Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:49:34.033977Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:49:34.034037Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:49:34.034155Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:49:44.040766Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:49:44.040850Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:49:44.040953Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:49:44.040966Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:49:54.047538Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:49:54.047889Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:49:54.047897Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:49:54.047919Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:04.054401Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:50:04.054548Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:50:04.054909Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:04.054958Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:50:14.061171Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:50:14.061177Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:50:14.061498Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:14.061723Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:50:24.068200Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:50:24.068205Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:50:24.068332Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:24.068383Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:50:34.075132Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:50:34.075158Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:50:34.075395Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:50:34.075418Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:44.082019Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:50:44.082018Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:50:44.082117Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:50:44.082167Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:54.088794Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:50:54.088847Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:50:54.089019Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:54.089022Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:04.095735Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:51:04.095776Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:51:04.096233Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:04.098523Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:51:14.102841Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:51:14.103346Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:14.103982Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:51:14.109104Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:51:24.109849Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:51:24.109873Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:24.110851Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:51:24.115700Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:51:34.116727Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:51:34.116823Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:34.117833Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:51:34.122757Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:51:44.123718Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:51:44.123914Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:44.124249Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:51:44.129360Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:51:54.130816Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:54.130863Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:51:54.130905Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:51:54.141376Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:52:04.137669Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:52:04.137673Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:52:04.137673Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:52:04.148239Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:52:14.144571Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:52:14.144582Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:52:14.144615Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:52:14.155153Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:52:24.151693Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:52:24.151697Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:52:24.151768Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:52:24.162123Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:52:34.158579Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:52:34.158597Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:52:34.158594Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:52:34.169015Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:52:44.165590Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:52:44.165597Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:52:44.165588Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:52:44.176320Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:52:54.172531Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:52:54.172595Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:52:54.172673Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:52:54.183247Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:53:04.179369Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:53:04.179443Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:53:04.179519Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:53:04.190149Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:53:14.186236Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:53:14.186278Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:53:14.186365Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:53:14.197106Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:53:24.193077Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:53:24.193127Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:53:24.193156Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:53:24.204008Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:53:34.199926Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:53:34.199971Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:53:34.200042Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:53:34.210785Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:53:44.208146Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:53:44.208955Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:53:44.208959Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:53:44.217787Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:53:54.215026Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:53:54.215744Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:53:54.215801Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:53:54.224443Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:54:04.222008Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:54:04.222203Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:54:04.222564Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:54:04.231264Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:54:14.228902Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:54:14.228891Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:54:14.229379Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:54:14.237779Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:54:24.235708Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:54:24.235840Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:54:24.236172Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:54:24.244499Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:54:34.242573Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:54:34.242677Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:54:34.242920Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:54:34.251412Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:54:44.249302Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:54:44.249442Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:54:44.249742Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:54:44.258099Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:54:54.256299Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:54:54.256324Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:54:54.256620Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:54:54.264803Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:55:04.263293Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:55:04.263385Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:55:04.263601Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:55:04.271516Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:55:14.270308Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:55:14.270316Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:55:14.270876Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:55:14.278237Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:55:24.277208Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:55:24.277305Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:55:24.277760Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:55:24.285046Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:55:34.284250Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:55:34.284269Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:55:34.284716Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:55:34.293369Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:55:44.291195Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:55:44.291428Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:55:44.291778Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:55:44.300106Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:55:54.298032Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:55:54.298093Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:55:54.298580Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:55:54.307091Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:56:04.305016Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:56:04.305224Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:56:04.305505Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:56:04.313885Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:56:14.312071Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:56:14.312118Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:56:14.312307Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:56:14.320593Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:56:24.319164Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:56:24.319192Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:56:24.319303Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:56:24.327472Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:56:34.326131Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:56:34.326166Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:56:34.326218Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:56:34.334405Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:56:44.333060Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:56:44.333060Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:56:44.333113Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:56:44.341175Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:56:54.339970Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:56:54.339970Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:56:54.340054Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:56:54.348120Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:57:04.346984Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:57:04.347012Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:57:04.347022Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:57:04.355152Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:57:14.353976Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:57:14.354024Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:57:14.354047Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:57:14.361882Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:57:24.360885Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:57:24.360888Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:57:24.360948Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:57:24.368766Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:57:34.367645Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:57:34.367676Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:57:34.367651Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:57:34.375281Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:57:44.374476Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:57:44.374575Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:57:44.374682Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:57:44.382112Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:57:54.381574Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:57:54.381651Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:57:54.382005Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:57:54.389292Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:58:04.388513Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:58:04.388531Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:58:04.388852Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:58:04.396262Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:58:14.395839Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:58:14.395849Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:58:14.395849Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:58:14.405708Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:58:24.402939Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:58:24.402936Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:58:24.402993Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:58:24.412681Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:58:34.409699Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:58:34.409953Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:58:34.410033Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:58:34.419615Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:58:44.416694Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:58:44.416867Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:58:44.416957Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:58:44.426414Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:58:54.423747Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:58:54.423892Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:58:54.424043Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:58:54.433169Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:59:04.430729Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:59:04.430914Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:59:04.431060Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:59:04.440080Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:59:14.437713Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:59:14.438082Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:59:14.438815Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:59:14.446831Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:59:24.444769Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:59:24.444794Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:59:24.445716Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:59:24.453586Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:59:34.451772Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:59:34.451788Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:59:34.452669Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:59:34.460941Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:59:44.458765Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:59:44.459001Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:59:44.459531Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:59:44.467846Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:59:54.465755Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:59:54.466185Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:59:54.466608Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:59:54.474791Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:00:04.472814Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:00:04.473111Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:00:04.473398Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:00:04.481512Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:00:14.479759Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:00:14.479792Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:00:14.480177Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:00:14.488542Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:00:24.486670Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:00:24.486754Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:00:24.486976Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:00:24.495513Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:00:34.493631Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:00:34.493644Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:00:34.493930Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:00:34.502467Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:00:44.500444Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:00:44.500477Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:00:44.500814Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:00:44.509281Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:00:54.507466Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:00:54.507614Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:00:54.507866Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:00:54.516409Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:01:04.514251Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:01:04.514577Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:01:04.514601Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:01:04.523495Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:01:14.521457Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:01:14.521624Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:01:14.521628Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:01:14.530334Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:01:24.528305Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:01:24.528480Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:01:24.528582Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:01:24.537176Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:01:34.535630Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:01:34.535625Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:01:34.535651Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:01:34.544026Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:01:44.542635Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:01:44.542668Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:01:44.542637Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:01:44.550866Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:01:54.549717Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:01:54.549767Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:01:54.549817Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:01:54.557846Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:02:04.556633Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:02:04.556679Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:02:04.556683Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:02:04.564676Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:02:14.563653Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:02:14.563700Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:02:14.563737Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:02:14.571837Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:02:24.570651Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:02:24.570673Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:02:24.570735Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:02:24.579002Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:02:34.577383Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:02:34.577570Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:02:34.577763Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:02:34.585797Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:02:44.584306Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:02:44.584314Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:02:44.584970Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:02:44.592416Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:02:54.591370Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:02:54.591447Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:02:54.592128Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:02:54.601378Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:03:04.598287Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:03:04.598395Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:03:04.598909Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:03:04.608307Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:03:14.605140Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:03:14.605228Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:03:14.605638Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:03:14.615124Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:03:23.863102Z  INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-0
2025-02-11T06:03:23.922983Z  INFO shard-manager: text_generation_launcher: Shard ready in 849.903057579s rank=0
2025-02-11T06:03:24.612319Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:03:24.612449Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:03:24.612487Z  INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:03:25.859857Z  INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-2
2025-02-11T06:03:25.860944Z  INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-3
2025-02-11T06:03:25.865287Z  INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-1
2025-02-11T06:03:25.913263Z  INFO shard-manager: text_generation_launcher: Shard ready in 851.892492758s rank=3
2025-02-11T06:03:25.913384Z  INFO shard-manager: text_generation_launcher: Shard ready in 851.893030923s rank=2
2025-02-11T06:03:25.913417Z  INFO shard-manager: text_generation_launcher: Shard ready in 851.893317076s rank=1
2025-02-11T06:03:25.980482Z  INFO text_generation_launcher: Starting Webserver
2025-02-11T06:03:26.687648Z  INFO text_generation_router: router/src/main.rs:228: Using the Hugging Face API
2025-02-11T06:03:26.687697Z  INFO hf_hub: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/hf-hub-0.3.2/src/lib.rs:55: Token file not found "/tmp/.cache/huggingface/token"    
2025-02-11T06:03:36.170416Z  INFO text_generation_router: router/src/main.rs:577: Serving revision 3865e12a1eb7cbd641ab3f9dfc28c588c6b0c1e9 of model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
2025-02-11T06:03:36.484596Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<｜end▁of▁sentence｜>' was expected to have ID '151643' but was given ID 'None'    
2025-02-11T06:03:36.484611Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<｜User｜>' was expected to have ID '151644' but was given ID 'None'    
2025-02-11T06:03:36.484614Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<｜Assistant｜>' was expected to have ID '151645' but was given ID 'None'    
2025-02-11T06:03:36.484617Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<｜begin▁of▁sentence｜>' was expected to have ID '151646' but was given ID 'None'    
2025-02-11T06:03:36.484618Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|EOT|>' was expected to have ID '151647' but was given ID 'None'    
2025-02-11T06:03:36.484620Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<think>' was expected to have ID '151648' but was given ID 'None'    
2025-02-11T06:03:36.484635Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '</think>' was expected to have ID '151649' but was given ID 'None'    
2025-02-11T06:03:36.484637Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|quad_start|>' was expected to have ID '151650' but was given ID 'None'    
2025-02-11T06:03:36.484638Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|quad_end|>' was expected to have ID '151651' but was given ID 'None'    
2025-02-11T06:03:36.484640Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|vision_start|>' was expected to have ID '151652' but was given ID 'None'    
2025-02-11T06:03:36.484641Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|vision_end|>' was expected to have ID '151653' but was given ID 'None'    
2025-02-11T06:03:36.484643Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|vision_pad|>' was expected to have ID '151654' but was given ID 'None'    
2025-02-11T06:03:36.484644Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|image_pad|>' was expected to have ID '151655' but was given ID 'None'    
2025-02-11T06:03:36.484646Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|video_pad|>' was expected to have ID '151656' but was given ID 'None'    
2025-02-11T06:03:36.484647Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<tool_call>' was expected to have ID '151657' but was given ID 'None'    
2025-02-11T06:03:36.484649Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '</tool_call>' was expected to have ID '151658' but was given ID 'None'    
2025-02-11T06:03:36.484650Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|fim_prefix|>' was expected to have ID '151659' but was given ID 'None'    
2025-02-11T06:03:36.484652Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|fim_middle|>' was expected to have ID '151660' but was given ID 'None'    
2025-02-11T06:03:36.484654Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|fim_suffix|>' was expected to have ID '151661' but was given ID 'None'    
2025-02-11T06:03:36.484658Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|fim_pad|>' was expected to have ID '151662' but was given ID 'None'    
2025-02-11T06:03:36.484660Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|repo_name|>' was expected to have ID '151663' but was given ID 'None'    
2025-02-11T06:03:36.484662Z  WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|file_sep|>' was expected to have ID '151664' but was given ID 'None'    
2025-02-11T06:03:36.486228Z  INFO text_generation_router: router/src/main.rs:342: Overriding LlamaTokenizer with TemplateProcessing to follow python override defined in https://github.com/huggingface/transformers/blob/4aa17d00690b7f82c95bb2949ea57e22c35b4336/src/transformers/models/llama/tokenization_llama_fast.py#L203-L205
2025-02-11T06:03:36.486235Z  INFO text_generation_router: router/src/main.rs:357: Using config Some(Qwen2)
2025-02-11T06:03:36.486239Z  WARN text_generation_router: router/src/main.rs:384: Invalid hostname, defaulting to 0.0.0.0
2025-02-11T06:03:36.547050Z  INFO text_generation_router::server: router/src/server.rs:1572: Warming up model
2025-02-11T06:03:40.722557Z  INFO text_generation_launcher: Cuda Graphs are enabled for sizes [0]
2025-02-11T06:03:40.823710Z  INFO text_generation_router::server: router/src/server.rs:1599: Using scheduler V3
2025-02-11T06:03:40.823736Z  INFO text_generation_router::server: router/src/server.rs:1651: Setting max batch total tokens to 373920
2025-02-11T06:03:40.920081Z  INFO text_generation_router::server: router/src/server.rs:1889: Connected
2025-02-11T06:04:26.630119Z  INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.285263251s" validation_time="1.036747ms" queue_time="119.465µs" inference_time="1.284107407s" time_per_token="33.7923ms" seed="Some(17207679548585787341)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:05:33.038265Z  INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.404232248s" validation_time="560.26µs" queue_time="52.421µs" inference_time="1.403619944s" time_per_token="31.900453ms" seed="Some(16275190732384590354)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:05:55.181522Z  INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.216876593s" validation_time="948.563µs" queue_time="51.145µs" inference_time="1.215877199s" time_per_token="31.996768ms" seed="Some(14699494694487423509)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:06:00.596233Z  INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.391676975s" validation_time="349.937µs" queue_time="58.445µs" inference_time="1.391268961s" time_per_token="31.619749ms" seed="Some(15828993800647044877)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:06:14.351810Z  INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.210583081s" validation_time="469.041µs" queue_time="113.129µs" inference_time="1.210001384s" time_per_token="31.842141ms" seed="Some(3690229400508701128)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:06:32.156409Z  INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.210452984s" validation_time="376.402µs" queue_time="42.773µs" inference_time="1.210034249s" time_per_token="31.843006ms" seed="Some(6777010132799794204)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:06:43.358804Z  INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.394022016s" validation_time="361.85µs" queue_time="37.92µs" inference_time="1.39362255s" time_per_token="31.673239ms" seed="Some(725893851467889474)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:08:20.017441Z  INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.211224469s" validation_time="379.444µs" queue_time="44.521µs" inference_time="1.210800824s" time_per_token="31.863179ms" seed="Some(5207544670684677685)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:08:55.509801Z  INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="21.24950047s" validation_time="764.322µs" queue_time="89.915µs" inference_time="21.248646533s" time_per_token="31.526181ms" seed="Some(14816016702781754406)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:09:48.768802Z  INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="27.378218083s" validation_time="452.25µs" queue_time="84.92µs" inference_time="27.377681266s" time_per_token="31.432469ms" seed="Some(896637297712218630)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:10:27.694657Z  INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="28.452501692s" validation_time="3.629787ms" queue_time="48.31µs" inference_time="28.448824118s" time_per_token="32.699797ms" seed="Some(16454489381862762319)"}: text_generation_router::server: router/src/server.rs:511: Success

Attachments

No response
@hualongfeng hualongfeng added the bug Something isn't working label Feb 11, 2025
@lianhao
Copy link
Collaborator

lianhao commented Feb 12, 2025

Please try 1.2 version of chatqna to see if this problem is still existing there

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lianhao @hualongfeng