Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

short write error keeps appearing in 1.8.5 while everything goes well in 1.6.6 #21712

Closed
chengshiwen opened this issue Jun 17, 2021 · 4 comments

Comments

@chengshiwen
Copy link

chengshiwen commented Jun 17, 2021

Steps to reproduce:
List the minimal actions needed to reproduce the behavior.

I don't know how to reproduce. If a measurement is dropped and a large amount of data of the measurement is written suddenly, it is possible

Expected behavior:
Describe what you expected to happen.
Bug fixed

Actual behavior:
Describe What actually happened.

image
image
image
image

Environment info:

  • System info: Run uname -srm and copy the output here
  • InfluxDB version: Run influxd version and copy the output here
  • Other relevant environment details: Container runtime, disk info, etc

InfluxDB 1.8.5, Docker

If I downgrade to 1.6.6, everything goes well

Config:
Copy any non-default config values here or attach the full config as a gist or file.

INFLUXDB_DATA_INDEX_VERSION=tsi1
INFLUXDB_DATA_WAL_FSYNC_DELAY=10ms
@chengshiwen chengshiwen changed the title error: short write keeps appearing short write error keeps appearing Jun 17, 2021
@chengshiwen chengshiwen changed the title short write error keeps appearing short write error keeps appearing in 1.8.5 while everything goes well in 1.6.6 Jun 18, 2021
@lifeibo
Copy link

lifeibo commented Jul 23, 2021

It maybe a bug while DropMeasurement.

--- a/tsdb/index/tsi1/log_file.go
+++ b/tsdb/index/tsi1/log_file.go
@@ -1121,8 +1121,6 @@ func (f *LogFile) seriesSketches() (sketch, tSketch estimator.Sketch, err error)
 }

 func (f *LogFile) ExecEntries(entries []LogEntry) error {
-       f.mu.Lock()
-       defer f.mu.Unlock()
        for i := range entries {
                entry := &entries[i]
                if err := f.appendEntry(entry); err != nil {
@@ -1134,6 +1132,9 @@ func (f *LogFile) ExecEntries(entries []LogEntry) error {
 }

 func (f *LogFile) Writes(entries []LogEntry) error {
+       f.mu.Lock()
+       defer f.mu.Unlock()
+
        if err := f.ExecEntries(entries); err != nil {
                return err
        }

	// Flush buffer and sync to disk.
	return f.FlushAndSync()

@chengshiwen
Copy link
Author

@lifeibo Yes! I did execute drop measurement first!

@altery
Copy link

altery commented Apr 25, 2022

We also experience these short write errors from time to time. Also, we had deletion of measurements shortly before the errors started to appear.
We have to restart influxdb to recover from that situation.

@altery
Copy link

altery commented Aug 10, 2022

Is it known, if influxdb 2.x is also affected by this problem?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants