change ReadChan to return receive-only chan #7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The concern is that this is an API-breaking change, and existing users will need to update, and will not be able to be compatible with both older and latest version at the same time. But hopefully there are very few users besides nsq, and hopefully all users lock to specific versions like nsq does. So, because I haven't seen any objections, and I want to lean towards letting the codebase keep moving, though at a slow pace ... I'll merge. |
|
(I ran the minimal tests in this repo locally) |
ploxiln
changed the title
ploxiln
changed the title
moredure
pushed a commit
to cloudlinux/go-diskqueue
that referenced
this pull request
Jan 10, 2023
* trivial improvement of some checks * use file instead of bufio.Reader in readNumOfMessages from file end Signed-off-by: Leon Ziyang Zhang <leonzz@google.com>
moredure
added a commit
to cloudlinux/go-diskqueue
that referenced
this pull request
Jan 10, 2023
* Keep track of the number of messages in the writeFile and the number of messages we read in the readFile. Also, reserve the last 4 bytes at the end of the file for the number of messages in that file. * Update tests to reflect changes made to the metadata file. Metadata file now contains readMessages and writeMessages. * Test thtat write messages resets when a new file is created. * Add a test that checks if meta data file is correct after writing a complete file and completely reading a file. * Allocate 8 bytes instead of 4 since writeMessages is in64, and reset writeMessages and readMessages in skipToNextRWFile. * Update comment about number of bytes reserved for writeMessages. * Allow user to choose whether to use new feature or not. * Separate testing from the original with the implementation of depth for disk space limit. * Revert to original code style. * Add comment for functions. * Explain that maxBytesDiskSpace is 0 when user is not using disk space limit feature. * Add explicit disk size limit feature flag. * Add a new line after if block. * Track the disk space the files tracked by DiskQueue takes up, and test that these numbers are accurate. * Update totalBytes and writePos when writing number of messages, and resolve the writeMessages off by 1 error. * Remove the additional 8 bytes to totalBytes. * Revert "Remove the additional 8 bytes to totalBytes." This reverts commit 652b548. * Revert "Update totalBytes and writePos when writing number of messages, and resolve the writeMessages off by 1 error." This reverts commit 48c7a85. * Revert changes that removed the additional 8 bytes. * Add comment to make code more readable. * Add test that completes the write file by meeting the file size limit exactly with a message less than 8 bytes. * Add extra testing to validate the increment/decrement of bytes in core code. * Update variable names. * Revert "Update variable names." This reverts commit e9b23ac. * Revert "Merge branch 'TrackDiskSize' into DepthImpl" This reverts commit 76a0ddd, reversing changes made to 30c6ef6. * Update variable names. * Replace 8 with a constant to improve readability. * Reset Diskqueue data during readError. * Track disk size (nsqio#5) * Remove readBytes and update testing. * Add comment for readMsgSize. * Modify disk size limit features only if disk size limit feature is being used . * Reset write bytes during read error if read file is write file. * Track writeBytes and readMsgSize. * Update test. * Test writeBytes is accurate. * Make code more readable. * Revert back to using msgSize instead of readMsgSize in order to make minimal changes. * Update max bad file size to include numFileMsgsBytes. * Update go.mod * Disk size limit (nsqio#6) * Remove readBytes and update testing. * Add comment for readMsgSize. * Modify disk size limit features only if disk size limit feature is being used . * Increase disk size limit in testing, and get MetaData file size. * Add comments to metaDataFileSize func for better readability. * Add logic to remove readFile if we are going to surpass the Disk Size Limit. * Abstract code from moveForward and create makeSpace function to be used when DiskQueue needs to make space. * Rename func to make it more readable. * Make space when the new writeMessage will surpass the disk size limit. Test that DiskQueue never surpasses the disk size limit. * Make space until there is enough space to write the message * Abstract code and make a reachFileSizeLimit flag. * Update testing to account for metadata file size. * Handle the deletion of .bad files if it exists. * Get oldest bad file info does not need to return an error. It either finds a bad file or does not. * Track size of bad files. * Decrease badBytes when we delete a bad file and move badBytes to check if we need to make disk space. * Remove badBytes from metadata and make it an atomic field. * Recalculate badBytes if it is a negative number. * Add comments and rename varaibles to improve readability. * Update metaData file size now that badBytes is removed from metaData file. * Prevent user from writing data that is bigger than disk size limit. * Make the code more readable by changing the order of checks in writeOne(). * Test the scenario when msgSize needs the deletion of several files. * Use regex when getting diskqueue files with .bad extension. * Update the tracking of .bad files to use regex. * Remove log messages. * Start writing a test to check if .bad files are deleted or accounted for correctly. * Test that DiskQueue deleted .bad files first in order to make Disk Space. * Update with to other branches - variable name change and adjust writeBytes, writeMsgs, and readMsgs in handReadError. * Adjust writeBytes as a result of the initial overestimation of the size of a bad file when we cannot get its accurate size. * Reset messages in handleReadError * Use MatchString as opposed to Match. * Throw error. * Instead of overestimating bad file size, underestime it. This ensures we will never go over disk limit. * Add a check to see if readFile is corrupted. * Remove DEBUG logs. * test .bad files. * In progress... corrupt file and then have readOne() deem it corrupted. * Make code more readable. * Ensure that diskqueue handles corrupted files correctly. * Remove badBytes and get the accurate writeBytes data every time a writeFile is deleted or a readFile turns into a bad file. * Abstract general function to walk through all of the files in the directory DiskQueue writes and reads in. * Abstract code to its own function. * Remove readMsgSize and make code more readable. * Make test code more readable. * Update go.mod * Make regexp constants and close metadatafile after getting its size. * Change writeBytes to totalDiskSpaceUsed and have it track writeBytes and badBytes for now. * Update panic messages. * Track the total disk size with one variable rather than several smaller ones. * Remove unnecessary comments. * Break up huge function into two smaller functions. * Ensure that global regExp are created on start. * Add testing of depth, add info logs, and test when disk size limit is too small (not enough space for meta data file). * Update name of dq objects to match function name. * Use ReadDir instead of WalkDir * Replace go v1.16 functions and objects with <v1.16 functions and objects. * Have metaData file size stay as 56 bytes and update testing. * Add the number of bytes that were removed when a file is removed. * Only get the size if the file was able to be removed successfully. * Add function to get filder size and update log when removing read file to make space. * Make the code more readable. * Extract chunk of code into a new function. * Improve readability of the code and update code from nsqio#29 issue 29 * totalDiskSpaceUsed does not change in handleReadError * Iterate through each file rather than relying on the totalDiskSpaceUsed variable. * Revert "Merge branch 'master' into DiskSizeLimit" This reverts commit 0c456b4, reversing changes made to a821855. * Update README.md * Update README.md Updated the Build status, Go reference, and latest Github release * Read correct number of messages from end of file (nsqio#7) * trivial improvement of some checks * use file instead of bufio.Reader in readNumOfMessages from file end Signed-off-by: Leon Ziyang Zhang <leonzz@google.com> * Update Readme and add description for the fork. * Update \d\d\d\d\d\d to \d+ * Fixed issue with shared regexps * Update go.mod * Fixed conflicting variable issue * Fixed issue with peekChan * Reverted switch to next file before maxBytesPerFile is reached * Fixed issue with .bad files * Added timeout between corruption checks * Added more timeout between corruption checks * Reduced timeouts * Timeout replaced * Fixed pattern of bad files * Ref * Check failures * Added error handling * Fixed tests Signed-off-by: Leon Ziyang Zhang <leonzz@google.com> Co-authored-by: Kevin Cam <kevincam@google.com> Co-authored-by: Kevin Cam <kevinsebcam@gmail.com> Co-authored-by: CatherineF-dev <78218824+CatherineF-dev@users.noreply.github.com> Co-authored-by: Leon Ziyang Zhang <leonzz@mikreal.com> Co-authored-by: Leon Ziyang Zhang <leonzz@google.com> Co-authored-by: Mikhail Faraponov <mfaraponov@cloudlinux.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ReadChan should return receive-only chan for reading.