btrfs-convert "no-holes": extent-tree.c:2764: alloc_tree_block: BUG_ON ret triggered, value -28
#123
Comments
denji
changed the title
ret triggered, value -28ret triggered, value -28
|
This means btrfs can't allocate new tree block due to no available space. （indicated by -ENOSPC). The good news is, all modification is out of used ext* data, so your fs should be still be completely safe. Unfortunately, we don't have a good way to know if there is enough free space of ext*. |
|
BTW, would you please provide the df output of that ext4 (mounted) fs? Current btrfs-convert can't handle fragmented free space of ext4 well, due to btrfs metadata/data chunk size limitation. If you really want to convert it to btrfs, I'd recommend to remove some files in that ext4 filesystem, to ensure enough continuous free space, or try to skip datasum to save some metadata space. |
|
This is no longer possible, the test cluster is already formatted. |
|
Just ran into this problem with btrfs-convert 4.17.1 when running it on a 1.3T ext4 filesystem. About 850G of the space was used... Am trying to do a defragment-via-resize2fs to see if that helps at all. |
|
/dev/md126p6 1.3T 821G 424G 66% /repo resize2fs down to 830G and back to 1.3T does not fix the problem. I also noticed that there are three messages about "Unable to find block group for 0" before the error. |
|
@elliotclee Great thanks for the info. If shrinked then appended doesn't work, it may be the problem of csum taking too many space while file tree operation doesn't get its chance to allocate new chunks. The "unable to find block group for 0" is just some error message unhanled for such case. Would you please convert without csum nor inline data? ( Or you could try this branch https://github.com/adam900710/btrfs-progs/tree/chunk_allocation |
|
Tried it with just -n and it still errored out. With -d -n it completes successfully. |
|
Still happens with btrfs-progs 4.17.1. In btrfs-progs 4.19.1 from Fedora Rawhide, it gives this error (which looks like the same thing with code that got refactored...) ctree.c:2244: split_leaf: BUG_ON |
|
When I ran 4.19.1 with -d, I got: ctree.c:2244: split_leaf: BUG_ON When I ran 4.19.1 with -d -n, I got: |
|
4.17.1 can do convert with just -d OK, so it seems the bug got even worse in 4.18 or 4.19. |
|
I've added Qu's fix to devel branch, scheduled for 5.0 release. |
… extent tree In github issues, one user reports unexpected ENOSPC error if enabling datasum druing convert. After some investigation, it looks like that during ext2_saved/image creation, we could create large file extent whose size can be 128M (max data extent size). In that case, its csum block will be at least 128K. Under certain case we need to allocate extra metadata chunks to fulfill such space requirement. However we only do metadata prealloc if we're reserving extents for fs trees. (we use btrfs_root::ref_cows to determine whether we should do metadata prealloc, and that member is only set for fs trees). There is no explaination on why we only do metadata prealloc for file trees, but at least from my investigation, it could be related to avoid nested extent tree modication. At least extent reservation for csum tree shouldn't be a problem with metadata block group preallocation. So change the metadata block group preallocation check from "root->ref_cow" to "root->root_key.objectid != BTRFS_EXTENT_TREE_OBJECTID", and add some comment for it. Issue: #123 Signed-off-by: Qu Wenruo <wqu@suse.com> Signed-off-by: David Sterba <dsterba@suse.com>
… extent tree In github issues, one user reports unexpected ENOSPC error if enabling datasum druing convert. After some investigation, it looks like that during ext2_saved/image creation, we could create large file extent whose size can be 128M (max data extent size). In that case, its csum block will be at least 128K. Under certain case we need to allocate extra metadata chunks to fulfill such space requirement. However we only do metadata prealloc if we're reserving extents for fs trees. (we use btrfs_root::ref_cows to determine whether we should do metadata prealloc, and that member is only set for fs trees). There is no explaination on why we only do metadata prealloc for file trees, but at least from my investigation, it could be related to avoid nested extent tree modication. At least extent reservation for csum tree shouldn't be a problem with metadata block group preallocation. So change the metadata block group preallocation check from "root->ref_cow" to "root->root_key.objectid != BTRFS_EXTENT_TREE_OBJECTID", and add some comment for it. Issue: #123 Signed-off-by: Qu Wenruo <wqu@suse.com> Signed-off-by: David Sterba <dsterba@suse.com>
In github issues, one user reports unexpected ENOSPC error if enabling datasum druing convert. After some investigation, it looks like that during ext2_saved/image creation, we could create large file extent whose size can be 128M (max data extent size). In that case, its csum block will be at least 128K. Under certain case we need to allocate extra metadata chunks to fulfill such space requirement. However we only do metadata prealloc if we're reserving extents for fs trees. (we use btrfs_root::ref_cows to determine whether we should do metadata prealloc, and that member is only set for fs trees). There is no explaination on why we only do metadata prealloc for file trees, but from my educated guess, it could be related to avoid nested extent/chunk tree modication. At least extent reservation for csum tree shouldn't be a problem with metadata block group preallocation. So adding new condition for metadata preallocate to avoid unexpected ENOSPC problem. Issue: kdave#123 Signed-off-by: Qu Wenruo <wqu@suse.com>
|
@elliotclee Would you please try with latest devel branch? And could you provide the image for us to reproduce? |
|
Hi, I just downloaded and compiled btrfs-progs from the devel branch as of this evening, and here's what a convert produced: [root@shiny btrfs-progs-devel]# ./btrfs-convert -L /dev/md126p6 I also ran it again under gdb and got this stack trace: (I also btrfs-convert with the -d -n flags, but it still errored out in btrfs_search_slot .) Unfortunately this filesystem is 840GB of files on a 1.3T partition, and even if I had a place to upload the image to, it would take 48 days at my current internet connection speed. I might be able to set up a time to provide temporary remote access to someone to debug it that way - let me know if that would be helpful. |
|
FWIW this issue shouldn't be closed, as it still occurs in btrfs-progs-5.1, even with -d -n. |
|
@elliotclee With your latest output, it still looks like it's csum tree causing the problem. For that case, we have patch to fix that (at least in theory). Would you please also show the calltrace using -d and -n? For the worst case, it's the ext* which has too fragmented free space that btrfs can't take much use of. It would provide great help if I can access the image. |
|
@adam900710 as mentioned in a previous comment, this fs is too large for me to be able to share it, but I can arrange to provide you with root access to the system if you want. I just did a resize2fs to defragment the free space. Here is the output of btrfs-convert under gdb: Thread 1 "btrfs-convert" hit Breakpoint 1, 0x00007ffff7d2176e in abort () from /lib64/libc.so.6 |
|
@adam900710 also as mentioned before, I can get convert with '-d -n' to work successfully with btrfs-progs 4.17.1, but starting with 4.19 even that errors out. |
|
@elliotclee Please try the following diff The diff will output the chunk and dev extents layout along with bg usage at 2 time points:
My initial guess is, after the initial ext2 image creation we're already running out of space, if so the diff would show that. Please note that, the output may be large, please redirect it and save the output. With the diff there is no need for gdb output. |
|
@adam900710 Here you go. |
|
It shows that during that ENOSPC, we have a running transaction which is large and never reached disk. This means:
For problem 1, I need to dig further to pin down the problem, but for 2. would you please try the following diff? (Can be applied separately along previous debug one) If it works, please also try without -d/-n to see if it solves the problem. Please note this is just a proof-of-concept modification, for the real fix we need to take a trade-off between performance and false ENOSPC. Thanks. |
|
Confirmed that after the introduction of delayed ref, we don't have reliable method to know if a block group is near full, so chunk preallocator never works for metadata block group. The root fix would go that direction. For now, we can commit more frequently as a fast workaround. |
|
@adam900710 The workaround seems to be...working, at least with -d -n flags :-) Is there any other information I can provide? |
|
@elliotclee No extra info needed for a while. As the root cause is located, I'll keep you updated for incoming fix. Thanks for all your feedback! |
[BUG] There is a bug report of unexpected ENOSPC from btrfs-convert. kdave#123 After some debug, even when we have enough unallocated space, we still hit ENOSPC at btrfs_reserve_extent(). [CAUSE] Btrfs-progs relies on chunk preallocator to make enough space for data/metadata. However after the introduction of delayed-ref, it's no longer reliable to relie on btrfs_space_info::bytes_used and btrfs_space_info::bytes_pinned to calculate used metadata space. For a running transaction with a lot of allocated tree blocks, btrfs_space_info::bytes_used stays its original value, and will only be updated when running delayed ref. This makes btrfs-progs chunk preallocator completely useless. And for btrfs-convert/mkfs.btrfs --rootdir, if we're going to have enough metadata to fill a metadata block group in one transaction, we will hit ENOSPC no matter whether we have enough unallocated space. [FIX] This patch will introduce btrfs_space_info::bytes_reserved to trace how many space we have reserved but not yet committed to extent tree. To support this change, this commit also introduces the following modification: - More comment on btrfs_space_info::bytes_* To make code a little easier to read - Export update_space_info() to preallocate empty data/metadata space info for mkfs. For mkfs, we only have a temporary fs image with SYSTEM chunk only. Export update_space_info() so that we can preallocate empty data/metadata space info before we start a transaction. - Proper btrfs_space_info::bytes_reserved update The timing is the as kernel (except we don't need to update bytes_reserved for data extents) * Increase bytes_reserved when call alloc_reserved_tree_block() * Decrease bytes_reserved when running delayed refs With the help of head->must_insert_reserved to determine whether we need to decrease. Issue: kdave#123 Signed-off-by: Qu Wenruo <wqu@suse.com>
|
@elliotclee Would you please try my latest branch to see if it solves your problem? If nothing wrong happens, it would be the root fix. Thanks. |
|
(This is with the workaround. Will try your latest patch in a minute.) Hmm, I didn't let it run to completion earlier. Now I ran into the following farther along when it was running: creating btrfs metadata #0 0x00007f6d32632eb5 in raise () from /lib64/libc.so.6 This happens at the place where it commits a transaction every so often when copying inodes. |
|
With your root-cause-fix, it works like a champ. While you're working in convert, can you look at the patch on /issues/175? It just speeds up the ext2 image creation step of convert and adds a progress indicator for it, mainly useful when data csumming is on. |
|
BTW, is that without any other modification just the root fix? If so, I think it's time for the fix to go upstream. And a quick glance into the unable to find ref bug, it is more complex than I think. I'll add extra upstream debug to see what we can do. |
|
Bleah, so the root-cause-fix only worked with -d -n. When I use it without any extra flags, I get: (gdb) where The output with your debugging patch is attached. |
In github issues, one user reports unexpected ENOSPC error if enabling datasum druing convert. After some investigation, it looks like that during ext2_saved/image creation, we could create large file extent whose size can be 128M (max data extent size). In that case, its csum block will be at least 128K. Under certain case we need to allocate extra metadata chunks to fulfill such space requirement. However we only do metadata prealloc if we're reserving extents for fs trees. (we use btrfs_root::ref_cows to determine whether we should do metadata prealloc, and that member is only set for fs trees). There is no explaination on why we only do metadata prealloc for file trees, but from my educated guess, it could be related to avoid nested extent/chunk tree modication. At least extent reservation for csum tree shouldn't be a problem with metadata block group preallocation. So adding new condition for metadata preallocate to avoid unexpected ENOSPC problem. Issue: kdave#123 Signed-off-by: Qu Wenruo <wqu@suse.com>
|
Oh, we still need one existing patch for that. The branch updated to include that patch, to allow csum tree modification to allocate new chunk. Sorry for the inconvenience, would you please fetch the latest branch and test again? Thanks. |
|
OK, with the latest, running with no flags, it completes the creation of the filesystem image, but hits that "unable to find ref" bug again when copying inodes. There are also some warnings about leaking reserved space: [root@shiny btrfs-progs-chunk_preallocator]# ./btrfs-convert -L /dev/md126p6 Stack trace: So I think issue #123 is probably fixed, but now that other bug is showing up. |
|
@elliotclee Would you please try the latest branch again? Thanks. |
|
Sorry, I can't give it another shot. I went ahead and started using the
filesystem by converting with -d, and it's not convenient to un-covert.
Thanks for all the fixes that got me this far.
Elliot
…On Mon, May 27, 2019, 12:20 AM Qu Wenruo ***@***.***> wrote:
@elliotclee <https://github.com/elliotclee> Would you please try the
latest branch again?
I've added a new debug output, so when you hit the problem again, at least
we could have some clue.
Thanks.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#123?email_source=notifications&email_token=AJNWQFGKDAY5YX4JAIOMFD3PXNORPA5CNFSM4E2M3U52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWIXNOQ#issuecomment-496072378>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AJNWQFGMTWND23BYQBFJT7TPXNORPANCNFSM4E2M3U5Q>
.
|
In github issues, one user reports unexpected ENOSPC error if enabling datasum druing convert. After some investigation, it looks like that during ext2_saved/image creation, we could create large file extent whose size can be 128M (max data extent size). In that case, its csum block will be at least 128K. Under certain case we need to allocate extra metadata chunks to fulfill such space requirement. However we only do metadata prealloc if we're reserving extents for fs trees. (we use btrfs_root::ref_cows to determine whether we should do metadata prealloc, and that member is only set for fs trees). There is no explaination on why we only do metadata prealloc for file trees, but from my educated guess, it could be related to avoid nested extent/chunk tree modication. At least extent reservation for csum tree shouldn't be a problem with metadata block group preallocation. So adding new condition for metadata preallocate to avoid unexpected ENOSPC problem. Issue: #123 Signed-off-by: Qu Wenruo <wqu@suse.com> Signed-off-by: David Sterba <dsterba@suse.com>
In github issues, one user reports unexpected ENOSPC error if enabling datasum druing convert. After some investigation, it looks like that during ext2_saved/image creation, we could create large file extent whose size can be 128M (max data extent size). In that case, its csum block will be at least 128K. Under certain case we need to allocate extra metadata chunks to fulfill such space requirement. However we only do metadata prealloc if we're reserving extents for fs trees. (we use btrfs_root::ref_cows to determine whether we should do metadata prealloc, and that member is only set for fs trees). There is no explaination on why we only do metadata prealloc for file trees, but from my educated guess, it could be related to avoid nested extent/chunk tree modication. At least extent reservation for csum tree shouldn't be a problem with metadata block group preallocation. So adding new condition for metadata preallocate to avoid unexpected ENOSPC problem. Issue: #123 Signed-off-by: Qu Wenruo <wqu@suse.com> Signed-off-by: David Sterba <dsterba@suse.com>
In github issues, one user reports unexpected ENOSPC error if enabling datasum druing convert. After some investigation, it looks like that during ext2_saved/image creation, we could create large file extent whose size can be 128M (max data extent size). In that case, its csum block will be at least 128K. Under certain case we need to allocate extra metadata chunks to fulfill such space requirement. However we only do metadata prealloc if we're reserving extents for fs trees. (we use btrfs_root::ref_cows to determine whether we should do metadata prealloc, and that member is only set for fs trees). There is no explaination on why we only do metadata prealloc for file trees, but from my educated guess, it could be related to avoid nested extent/chunk tree modication. At least extent reservation for csum tree shouldn't be a problem with metadata block group preallocation. So adding new condition for metadata preallocate to avoid unexpected ENOSPC problem. Issue: #123 Signed-off-by: Qu Wenruo <wqu@suse.com> Signed-off-by: David Sterba <dsterba@suse.com>
[BUG] There is a bug report of unexpected ENOSPC from btrfs-convert, issue #123. After some debugging, even when we have enough unallocated space, we still hit ENOSPC at btrfs_reserve_extent(). [CAUSE] Btrfs-progs relies on chunk preallocator to make enough space for data/metadata. However after the introduction of delayed-ref, it's no longer reliable to rely on btrfs_space_info::bytes_used and btrfs_space_info::bytes_pinned to calculate used metadata space. For a running transaction with a lot of allocated tree blocks, btrfs_space_info::bytes_used stays its original value, and will only be updated when running delayed ref. This makes btrfs-progs chunk preallocator completely useless. And for btrfs-convert/mkfs.btrfs --rootdir, if we're going to have enough metadata to fill a metadata block group in one transaction, we will hit ENOSPC no matter whether we have enough unallocated space. [FIX] This patch will introduce btrfs_space_info::bytes_reserved to track how many space we have reserved but not yet committed to extent tree. To support this change, this commit also introduces the following modification: - More comment on btrfs_space_info::bytes_* To make code a little easier to read - Export update_space_info() to preallocate empty data/metadata space info for mkfs. For mkfs, we only have a temporary fs image with SYSTEM chunk only. Export update_space_info() so that we can preallocate empty data/metadata space info before we start a transaction. - Proper btrfs_space_info::bytes_reserved update The timing is the as kernel (except we don't need to update bytes_reserved for data extents) * Increase bytes_reserved when call alloc_reserved_tree_block() * Decrease bytes_reserved when running delayed refs With the help of head->must_insert_reserved to determine whether we need to decrease. Issue: #123 Signed-off-by: Qu Wenruo <wqu@suse.com> Signed-off-by: David Sterba <dsterba@suse.com>
[BUG] There is a bug report of unexpected ENOSPC from btrfs-convert, issue #123. After some debugging, even when we have enough unallocated space, we still hit ENOSPC at btrfs_reserve_extent(). [CAUSE] Btrfs-progs relies on chunk preallocator to make enough space for data/metadata. However after the introduction of delayed-ref, it's no longer reliable to rely on btrfs_space_info::bytes_used and btrfs_space_info::bytes_pinned to calculate used metadata space. For a running transaction with a lot of allocated tree blocks, btrfs_space_info::bytes_used stays its original value, and will only be updated when running delayed ref. This makes btrfs-progs chunk preallocator completely useless. And for btrfs-convert/mkfs.btrfs --rootdir, if we're going to have enough metadata to fill a metadata block group in one transaction, we will hit ENOSPC no matter whether we have enough unallocated space. [FIX] This patch will introduce btrfs_space_info::bytes_reserved to track how many space we have reserved but not yet committed to extent tree. To support this change, this commit also introduces the following modification: - More comment on btrfs_space_info::bytes_* To make code a little easier to read - Export update_space_info() to preallocate empty data/metadata space info for mkfs. For mkfs, we only have a temporary fs image with SYSTEM chunk only. Export update_space_info() so that we can preallocate empty data/metadata space info before we start a transaction. - Proper btrfs_space_info::bytes_reserved update The timing is the as kernel (except we don't need to update bytes_reserved for data extents) * Increase bytes_reserved when call alloc_reserved_tree_block() * Decrease bytes_reserved when running delayed refs With the help of head->must_insert_reserved to determine whether we need to decrease. Issue: #123 Signed-off-by: Qu Wenruo <wqu@suse.com> Signed-off-by: David Sterba <dsterba@suse.com>
The text was updated successfully, but these errors were encountered: