[HELP] Thingsboard gateway file type storage issue

[sobehard]

Hi, I'm having issue with the file type storage in thingsboard gateway setup. The picture below show the storage log file of the thingsboard gateway. Previously I was using memory type storage setup, and it is working fine for storing offline data and upload all the offline data file when it is back to online. But when I change it to file type storage, there is an issue exist after running for some amount of time. At first, the event storage writer and event storage reader is working fine as there is data storing in the file and pass to the thingsboard server. But after awhile (due to the max_record_setting and max_read_record_count setup), it only perform file writing without read the file. At this time, the thingsboard gateway keep writing new file without passing the incoming data to the thingsboard server.

I was testing on version 2.2.3.2 and 2.2.4.2, both of the version having same issue. May I know how to solve this issue?

Version

• OS: Ubuntu 18.04
• Thingsboard IoT Gateway version 2.2.3.2 & 2.2.4.2
• Python version 3

[imbeacon]

HI @sobehard

Thank you for your interest in ThingsBoard IoT gateway and feedback, I will investigate this and fix it.

[sobehard]

HI @sobehard

Thank you for your interest in ThingsBoard IoT gateway and feedback, I will investigate this and fix it.

Hi, there is another bug in the lastest version of thingsboard gateway that I forgot to mention earlier. In version 2.2.4.2, the thingsboard gateway does not auto reconnect back to the thingsboard server when it is disconnected from the server (due to internet issue / server down).

[imbeacon]

@sobehard

I fixed the issue with auto reconnect, could you try to install version from sources and test it in your environment?

[sobehard]

@sobehard

I fixed the issue with auto reconnect, could you try to install version from sources and test it in your environment?

Alright, will try it out.

[sobehard]

@sobehard

I fixed the issue with auto reconnect, could you try to install version from sources and test it in your environment?

Already tested with the new update, the re-connection issue is resolved. Thanks.

[imbeacon]

@sobehard

Is it resolve the issue with storage read/write?

[sobehard]

@sobehard

Is it resolve the issue with storage read/write?

Nope, the issue is still there with version 2.2.4.2. Below is the screenshot of the file storage test.

[sobehard]

@ZBeacon
Sorry, I think I misunderstand what you mean install version from source. I was using pip3 install to upgrade the thingsboard version to 2.2.4.2 for testing. I getting the same result as the re-connection issue appear again (only work for first time re-connection). May I know how can I test the latest code?

[imbeacon]

@sobehard

To install the latest version, please use this guide.

[sobehard]

@ZBeacon
Alright, will test it out. Thanks for fast response.

[sobehard]

@ZBeacon

Hi, I had tested the latest version based on the instruction given. The reconnection to server issue did resolved by the latest version. But the file storage issue is still exist.

Based on the screen shot above, I was running the gateway in file storage configuration. The file system was working fine for first few minutes. After it reached the limit of the storage setting (max_read_record_count), it won't cleanup old file but keeping writting new file without update to the thingsboard server.

Screenshot above is generated file from the file storage configuration.

[imbeacon]

@sobehard

Please try now, storage should works properly.

[skangmy]

Hi @ZBeacon ,

I'm testing it with Windows 10. The gateway is sending data continuously right now.

But I notice it does not clean up old files due to the following errors:

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: './data/data_1587088710457.txt'

This leads to plenty of old files in the data folder and I need to delete manually them.

I'm not sure if it only happens in Windows. Will test on Ubuntu and feedback further

[sobehard]

Hi @ZBeacon

Sorry for the slow response. I just tested the latest commit source code from github. But I found that the storage issue still the same. Not sure why. Below is the screenshot of the storage.log file and content of the file inside the data path.

The setting that I used for the test is as shown in picture below.

[sobehard]

Hi @ZBeacon

I did test with windows 10 wls (which is same as @skangmy ), the file storage thing is working fine, the old file is deleted when a new file is created as shown in image below.

I was using the same file storage setting as ubuntu version as shown below.

But I found another issue with the latest commit code in windows 10 wls environment. I tried to disconnect the tb gateway from the thingsboard for certain amount of period then connect back. The offline file is created when there is more and more data coming in during the period. But the tb gateway seem like does not read all the file inside the folder during the "disconnected" period. When I reconnect the tb gateway back to thingsboard, it oni read and delete one file instead of whole bunch of file inside the folder. And there is no upcoming data from the tb gateway to thingsboard after that.

While the new file is keep created for incoming data without sending to thingsboard.

I had tried to restart the tb gateway, and it did print out Initializing from state file..., but it will only keep on creating new file without reading it and pushing to thingsboard. The only method that
can solved this issue temporary is to delete the "data" (storage) folder and let the tb gateway create a new folder.

I'm not sure whether I'm the one only have this issue, please correct me if I'm wrong.

ps: this is issue won't happen when there is only one file inside the "data" folder

[imbeacon]

Hi @sobehard

We have released a version 2.2.5. Please try it (or just download version from master branch). It should solve the issue.

[sobehard]

Hi @sobehard

We have released a version 2.2.5. Please try it (or just download version from master branch). It should solve the issue.

Hi,

Alright, will test it out and let you know the result.

[sobehard]

@ZBeacon

I had tested out the new release of thingsboard-gateway (version 2.2.5) yesterday. I did found some minor issue for the deamon process, will describe later. The file storage issue did resolved. Picture below is the result of testing.

Create offline file when disconnected from internet and cleaning old offline while it is back to online (data is uploaded to thingsboard at the same time

Stress test file storage for more than 6 hours

Above test was conducted using command "sudo thingsboard-gateway", I was having issue with the tb gateway when running in deamon process. I tried to test again with command "sudo systemctlm restart thingsboard-gateway" just now. The reconnection issue occur and the file storage thing also the same. Not sure why.

Once I switch back to command "sudo thingsboard-gateway", the above issue disappear. But it comes back again when I use "sudo systemctl restart thingsboard-gateway" to trigger the tb gateway.

[imbeacon]

Hi @sobehard

Please try the following command to update the gateway version for thingsboard_gateway user (It uses to run the gateway as a daemon):

sudo pip3 install thingsboard-gateway --user thingsboard_gateway --upgrade

[sobehard]

@ZBeacon

Hi, I already update the gateway version using the command that you stated "sudo pip3 install thingsboard-gateway --user thingsboard_gateway --upgrade'. Following is the screenshot of the update process in command window.

I having another error as shown below when running tb gateway as deamon after the update.

While when I run as "sudo thingsboard-gateway", the tb gateway working fine as shown below.

Following is screenshot of the user permission of tb gateway folder.

[sobehard]

@ZBeacon

Hi, do you have idea on how to solve the previous issue that I mention? Or I need to open a new issue for this? Btw, the following is the log file from tb gateway when run as daemon.

[imbeacon]

Hi @skangmy
I apologize for the long answer, it is an issue with error coding, to solve it please upgrade your version to the version from the development branch and try again.
Also please change degub level from INFO to DEBUG in /etc/thingsboard_gateway/config/logs.conf file. It will show to you more information.

[sobehard]

Hi @ZBeacon

Any solution for my issue? I had re-tried the tb gateway (version 2.2.5) today and I found out another issue. (Noted: test is conducted not with daemon process, but with sudo thingsboard-gateway as the daemon process still having issue now) I had start the tb gateway since morning, everything is working fine, storage file keep updating by deleting old file. After running for 6 hours +, the following error comes out suddenly (it is the content of the tb gateway storage log file).

I had checked the file inside the storage folder, but there is only one file. Not sure why it say too many open files. To temporary solve this, I had to manually delete the storage file and run the tb gateway again.

[sobehard]

Hi @ZBeacon

Any solution for my issue? I had re-tried the tb gateway (version 2.2.5) today and I found out another issue. (Noted: test is conducted not with daemon process, but with sudo thingsboard-gateway as the daemon process still having issue now) I had start the tb gateway since morning, everything is working fine, storage file keep updating by deleting old file. After running for 6 hours +, the following error comes out suddenly (it is the content of the tb gateway storage log file).

I had checked the file inside the storage folder, but there is only one file. Not sure why it say too many open files. To temporary solve this, I had to manually delete the storage file and run the tb gateway again.

@ZBeacon please ignore this issue as I think it is caused by the read/write limitation of ubuntu system itself.

[skangmy]

Hi @ZBeacon

I have tested version 2.2.5 recently. I noticed there files being created and deleted constantly and at a very fast pace but no data is being sent to thingsboard server.

The following is the content of the log:

This is the storage setting in tb_gateway.yaml:

storage: 
  type: file
  data_folder_path: ./data/
  max_file_count: 10
  max_read_records_count: 5000
  max_records_per_file: 1000

Any idea what have gone wrong?

[imbeacon]

Hi @skangmy

Please try the version from the develop/2.4-python branch. And let me know about results.

[skangmy]

Hi @ZBeacon

I have tested develop/2.4-python branch and it is still having issues. I created a test script that publishes data for 5 devices every second via MQTT. When I first run tb gateway, the data is being sent to thingsboard server every second. But once it starts cleaning up old data files, the data is only being sent every 20 seconds. Please refer to attached screenshot

The following is my storage setting:

storage:
  type: file
  data_folder_path: ./data/
  max_file_count: 100
  max_read_records_count: 100
  max_records_per_file: 100

[imbeacon]

Hi @skangmy

I created the same script to sent data
I cannot repeat the issue :(

Could you email me ibarkov@thingsboard.io, access credentials to investigate the issue in your environment? SSH/TeamViewer/etc.

[sobehard]

Hi @ZBeacon,

I had tested the new tb gateway (version 2.3.0.1) with file storage setup. Below is the screenshot of the thingsboard gateway log file.

I found that the storage files always replaced with same name and only the position of the file keep changing. The data from the storage file did pass to the thingsboard server, but after 50 minutes +, the thingsboard gateway stop publish the data to the server anyone. Below is the screenshot of the thingsboard dashboard for the testing.

Picture below shows the logging of the content of thingsboard gateway's static file when the thingsboard gateway stop sending data to thingsboard server.

When I restart the thingsboard gateway, it will start to push data to the server but the same issue will still happen after 50 minutes +. Can I know how to solve this?

Version

OS: Ubuntu 18.04
Thingsboard IoT Gateway version 2.3.0.1
Python version 3

[skangmy]

Hi @skangmy

I created the same script to sent data
I cannot repeat the issue :(

Could you email me ibarkov@thingsboard.io, access credentials to investigate the issue in your environment? SSH/TeamViewer/etc.

Hi @ZBeacon , sorry for the delay. I have sent you the access credentials to your email. As I was doing the setup and testing, I noticed the behavior has changed and it's pretty much similar to what @sobehard has reported.

[imbeacon]

Hi @skangmy, @sobehard

Thanks @skangmy , I saw the email, but I will be available to connect in few days.

@sobehard , @skangmy

I made some changes to the file storage and put them in the development branch, please check the version from the development branch. I am still testing this version, but it looks like it works.

[sobehard]

Thanks @ZBeacon, the latest version of tb gateway did solved the issue. Sorry for the slow response.

[wenyangz]

this issue still exist with the new gateway. please help to check.

[sferguson-ISI]

I'm also getting this error on the latest release (3.1) when using the "file" option instead of the memory option...