-
Notifications
You must be signed in to change notification settings - Fork 277
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
A better way to approximate SDEF interpolation using blender armatures #162
Comments
I think this might be a fruitful discussion, although I'm afraid I don't have a lot to offer except gripes. As somebody who's more of an exporter, I'm really grateful to powroupi for implementing storage of SDEF, but on my most recent model, I find that this is one of the issues where I still run it through a PMXE pass to enable SDEF, because I want to preserve quads, Blender materials. sharp edges, edge crease in my main .blend, and because Blender's handling of shapekeyed meshes is so painful. (The other issue where I need to run it through PMXE is material order, but that may be a function just of my workflow; and of course Blender mat view is nothing like MMD's, but I don't think that a fix for that is particularly viable, it's still faster to run through PMXE than to emulate MMD rendering in Cycles.) However, I'm still surprised that just enabling "preserve volume" on your main armature modifier isn't good enough. My understanding of preserve volume is that it's most similar to MMD's quaternion deformation, functional only in MMM. Sometimes SDEF is just what the doctor ordered, especially for knobby elbow bones.... There have been some SDEF papers, and I believe powroupi linked me one some time earlier, but I didn't really understand it. I imagine that implementing SDEF in Blender would be possible, but probably not the best use of one's limited time on Earth. |
SDEF data was discussed in #23 (comment), the reference provided by @nagadomi is broken, fortunately translation is there, and it's easy to search for SDEF papers. :) |
I was thinking about solving this issue after the previous PR. But I haven't even started it yet. Edit: var bone0 = bones[(int)v.BlendIndices[0]];
var bone1 = bones[(int)v.BlendIndices[1]];
var center = v.SdefC; // Vector3
var weight0 = v.BlendWeight[0]; // float
var weight1 = 1 - weight0; // float
var mat = Matrix.RotationQuaternion(Quaternion.Lerp(Quaternion.Identity, bone1.LocalRotation, weight1) * Quaternion.RotationMatrix(bone0.AbsoluteTransform)); // Matrix
var pos = v.Position; // Vector3
pos = Vector3.TransformCoordinate(pos - center, mat)
+ (center
+ (Vector3.TransformCoordinate(center, bone0.AbsoluteTransform) - center + Vector3.TransformCoordinate(v.SdefR0, bone0.AbsoluteTransform)) * weight0
+ (Vector3.TransformCoordinate(center, bone1.AbsoluteTransform) - center + Vector3.TransformCoordinate(v.SdefR1, bone1.AbsoluteTransform)) * weight1) * 0.5f;
v.Normal = Vector3.TransformNormal(v.Normal, mat);
v.Position = pos;
// WTFPL |
Thanks for all the replies! @nathanvasil Last time I exported a model, material/bone order seemed to export correctly. Though I have to admit, I don't like exporting because it always ends with me swearing way too much. @powroupi @nagadomi Thanks for the references! I will have a look at them and try to understand them. |
So why are there two radiuses? |
@Hogarth-MMD According to the code nagadomi posted, there is one radius per bone. Center is transformed by both bones, r0 is only transformed by bone0 and r1 is only transformed by bone1. Then a bunch of arithmetic happens to get the final vertex position. |
If I remember correctly, in bmesh, vertices can have custom properties. SDEF drivers on vertex custom properties? The animation nodes add-on uses bpy.app.handlers. I am not really sure how necessary SDEF is. Probably a lot of work would be needed to implement it in Blender. |
Can SDEF be created on vertices in PMX editor? Or how is that done? |
Yup, that's how it's done. You select one or more vertices and change their weight mode to SDEF (from edit/weights./SDEF.) PMXE figures out the centers from the bone positions. Editing centers can be done on a vert-by-vert basis but it's easier to move the bones and recalculate centers. |
Is there a test PMX model which shows how SDEF works in the simplest, clearest way, with the minimum number of vertices? This would be helpful to understand and implement SDEF in Blender. |
@Hogarth-MMD Regarding the necessity, it would be a feature that adds consistency to how a model is rendered. It's easy to achieve the same by manually changing weights for quaternion interpolation, but when a model is well weighted for SDEF, it's a shame not being able to use it. Another task where it would be helpful is to actually create SDEF weights in Blender and get an accurate preview that matches MMD. Regarding models with SDEF, Kakomiki's Aria comes to mind. I know that the AB8 161007_2 version has good SDEF weights, despite using extreme radius settings that I cannot understand. The AB8b remake probably uses SDEF too, but I'm not familiar with that model. Both versions are available here: https://onedrive.live.com/?authkey=%21AGzT%5F4WgGZYwEF4&id=BFE73CA818CE632F%21138&cid=BFE73CA818CE632F |
@Hogarth-MMD Here, http://www.mediafire.com/file/7z1qm63qb46o8z8/sdef.pmx/file , three cylinders, one at SDEF, one at BDEF, one at SDEF with altered centers. Hope it's the right balance of simple but not too simple. |
But a pair of bones would not normally be aligned with each other along the same straight line, like you see in this diagram. |
I tried to extract the SDEF data from the 3 SDEF shape keys, using the test model of @nathanvasil . In PMX editor, when a vertex has no SDEF, its SDEF values are (0,0,0), but in Blender I did not see any (0,0,0) values in the SDEF shape keys. |
import bpy
mmd_sdef_c = []
mmd_sdef_r0 = []
mmd_sdef_r1 = []
shape_keys_names = bpy.context.active_object.data.shape_keys.key_blocks.keys()
if "mmd_sdef_c" in shape_keys_names:
for v in bpy.context.active_object.data.shape_keys.key_blocks["mmd_sdef_c"].data:
mmd_sdef_c.append(v.co[:])
if "mmd_sdef_r0" in shape_keys_names:
for v in bpy.context.active_object.data.shape_keys.key_blocks["mmd_sdef_r0"].data:
mmd_sdef_r0.append(v.co[:])
if "mmd_sdef_r1" in shape_keys_names:
for v in bpy.context.active_object.data.shape_keys.key_blocks["mmd_sdef_r1"].data:
mmd_sdef_r1.append(v.co[:])
print("\n\n")
print("C SDEF data", len(mmd_sdef_c), "vertices")
print("\n")
print(mmd_sdef_c)
print("\n\n")
print("R0 SDEF data", len(mmd_sdef_r0), "vertices")
print("\n")
print(mmd_sdef_r0)
print("\n\n")
print("R1 SDEF data", len(mmd_sdef_r1), "vertices")
print("\n")
print(mmd_sdef_r1) |
Above is my python code to extract the SDEF data from the 3 SDEF shape keys, then store this data in 3 python lists and print this data to the Blender system console. Apart from that, I basically have no |
If you want to use SDEF data, here is a sample code: import bpy
obj = bpy.context.active_object # select the mesh object of a MMD model before running this script
pose_bones = obj.parent.pose.bones # its parent should be the armature object
key_blocks = obj.data.shape_keys.key_blocks
sdefC = key_blocks.get('mmd_sdef_c')
sdefR0 = key_blocks.get('mmd_sdef_r0')
sdefR1 = key_blocks.get('mmd_sdef_r1')
for v, c, r0, r1 in zip(obj.data.vertices, sdefC.data, sdefR0.data, sdefR1.data):
if v.co != c.co:
sdef_c = c.co
sdef_r0 = r0.co
sdef_r1 = r1.co
# now you get sdef data (sdef_c, sdef_r0, sdef_r1) and original vertex (v.co) Reference: |
So if v.co == c.co: then the vertex is not an SDEF vertex. Right? |
I can't do anything with the code quoted by @nagadomi . I would either need someone to translate it into Blender python, or explain it very clearly in plain English (maybe with images). |
I think you might have an easier time if you change the way that you think about bones. A bone is a center of rotation and scaling, and a center is a point, not a line. (Translation doesn't need a center.) The tail doesn't matter-- Blender uses it to establish the local Y axis, but that's arbitrary, and any set of orthogonal, non-zero local basis vectors could be used to determine the same global space transformation just as easily. And of course Blender can use the tail as a pocket of extra information for use with constraints. But those aren't essential qualities for a bone, and they don't have anything to do with the global space, post-constraint, vertex-deforming transformation of the bone. Tails are as unimportant as a bone's name for that. So a bone isn't really a line, it's a point. Because of that, every pair of bones can be seen as 'aligned', since there exists a straight line between any two points. Just offering in case that's useful to you. |
https://gist.github.com/nagadomi/aa39745ae6716b50c2a60288b093d14b |
To start with, I suggest forget about R0 and R1 temporarily. Just get the interpolation of vertex locations along geodesics relative to point C with mathutils slerp (spherical linear interpolation of vectors). That aspect should be just simple Vector mathematics. |
From the small amount of available information about SDEF, it seems to me that the location of point C of a vertex is being influenced by the conflicting influences of 2 bones. As these 2 bones rotate or move, how does this change the location of point C? |
I've implemented SDEF on blender python. You can try it with the following addon. In the code I posted above, the preprocessing of R0 and R1 is missing. It is necessary for models with extreme R0/R1 settings like AB8. # preprocessing that was missing
rc = r0 * weight0 + r1 * weight1
r0 = c + r0 - rc
r1 = c + r1 - rc
# SDEF
mat0 = bone0.matrix * bone0.bone.matrix_local.inverted()
rot0 = mat0.to_quaternion().normalized()
mat1 = bone1.matrix * bone1.bone.matrix_local.inverted()
rot1 = mat1.to_quaternion().normalized()
c_b0 = mat0 * c
c_b1 = mat1 * c
r0 = mat0 * r0
r1 = mat1 * r1
b = r0 * weight0 + r1 * weight1
loc = ((c + (c_b0 - c ) * weight0 + (c_b1 - c) * weight1) + b) * 0.5
mat = rot0.slerp(rot1, weight1).to_matrix().to_4x4()
pos = (mat * (vertex_co - c)) + loc Edit: |
@nagadomi Here is a simplified version: (assuming w0 + w1 == 1) 😄 def mmd_sdef_driver_function(shapekey, obj_name):
...
def calc_skin_mat(pose_bone):
if pose_bone.name not in memo:
mat = pose_bone.matrix * pose_bone.bone.matrix_local.inverted()
v = (mat, mat.to_3x3())
memo[pose_bone.name] = v
return v
else:
return memo[pose_bone.name]
for i, w0, w1, bone0, bone1, pos_c, cr0, cr1 in g_mmd_sdef_verts[obj.name]:
mat0, rot0 = calc_skin_mat(bone0)
mat1, rot1 = calc_skin_mat(bone1)
shapekey.data[i].co = rot0.lerp(rot1, w1)*pos_c + mat0*cr0*w0 + mat1*cr1*w1
def mmd_sdef_find_vertices(obj):
... ... ... ...
vertices.append((
i, w0, w1, bones[0]["pose_bone"], bones[1]["pose_bone"],
vd[i].co-c, (c+r0)/2, (c+r1)/2)) |
https://gist.github.com/nagadomi/aa39745ae6716b50c2a60288b093d14b |
Congratulations @nagadomi !! I think that the SDEF algorithm could be used to create joint-controlled morphs. DAZ\Poser models use joint-controlled morphs for the deformation of joints of DAZ\Poser models. Instead of a dynamically updated shape key, there would be permanent shape keys on the model which have shape key drivers, the shape keys being driven by bone rotations. Joint-controlled morphs would not have any issues with slow speed. I don't really know if this would be better or worse than your current method. I'm just throwing it out as an idea. |
In Blender terminology, a "joint-controlled morph" is called a "corrective shape key". |
@Hogarth-MMD |
I think that a good approximation method is to calculate the weight for two armature modifiers(linear and quaternion interpolation) that is the most similar deformation to SDEF, mentioned by @Takuyax. Now that SDEF is revealed, its weight can be optimized. |
Comparing with np version, non-np version run faster on my PC. 😕 def mmd_sdef_driver_function(shapekey, obj_name):
obj = bpy.data.objects[obj_name]
if mmd_sdef_check_binding(obj):
pass
import time
st = time.time()
#mmd_sdef_driver_function_0(shapekey, obj_name)
mmd_sdef_driver_function_np(shapekey, obj_name)
print(time.time()-st)
return 1.0
def mmd_sdef_driver_function_0(shapekey, obj_name):
shapekey_data = shapekey.data
for bone0, bone1, vid, w0, w1, pos_c, cr0, cr1 in g_mmd_sdef_verts[obj_name].values():
mat0 = bone0.matrix * bone0.bone.matrix_local.inverted()
mat1 = bone1.matrix * bone1.bone.matrix_local.inverted()
rot0 = mat0.to_3x3()
rot1 = mat1.to_3x3()
for vi, w0i, w1i, posi, cr0i, cr1i in zip(vid, w0, w1, pos_c, cr0, cr1):
shapekey_data[vi].co = rot0.lerp(rot1, w1i) * posi + mat0 * cr0i * w0i + mat1 * cr1i * w1i
def mmd_sdef_driver_function_np(shapekey, obj_name):
shapekey_co = np.zeros(len(shapekey.data) * 3, dtype=np.float32)
shapekey.data.foreach_get("co", shapekey_co)
shapekey_co = shapekey_co.reshape(len(shapekey.data), 3)
for bone0, bone1, vid, w0, w1, pos_c, cr0, cr1 in g_mmd_sdef_verts[obj_name].values():
mat0 = bone0.matrix * bone0.bone.matrix_local.inverted()
mat1 = bone1.matrix * bone1.bone.matrix_local.inverted()
rot0 = mat0.to_3x3()
rot1 = mat1.to_3x3()
shapekey_co[vid] = [rot0.lerp(rot1, w1[i]) * pos_c[i] + mat0 * cr0[i] * w0[i] + mat1 * cr1[i] * w1[i] for i in range(len(vid))]
shapekey.data.foreach_set("co", shapekey_co.reshape(3 * len(shapekey.data))) Minor changes for non-np version 2: def mmd_sdef_driver_function_v2(shapekey, obj_name):
shapekey_data = shapekey.data
for bone0, bone1, sdef_data in g_mmd_sdef_verts[obj_name].values():
mat0 = bone0.matrix * bone0.bone.matrix_local.inverted()
mat1 = bone1.matrix * bone1.bone.matrix_local.inverted()
rot0 = mat0.to_3x3()
rot1 = mat1.to_3x3()
for vid, w0, w1, pos_c, cr0, cr1 in sdef_data:
shapekey_data[vid].co = rot0.lerp(rot1, w1) * pos_c + mat0 * cr0 * w0 + mat1 * cr1 * w1
def mmd_sdef_find_vertices(obj):
... ... ... ...
if key not in vertices:
vertices[key] = (bones[0]["pose_bone"], bones[1]["pose_bone"], [])
vertices[key][2].append((i, w0, w1, vd[i].co-c, (c+r0)/2, (c+r1)/2))
def mmd_sdef_bind(obj):
...
mask = tuple(i[0] for v in g_mmd_sdef_verts[obj.name].values() for i in v[2]) |
Please forgive me if this is a dumb question, but what are np and non-np? |
@Hogarth-MMD speed up with and without numpy. Function |
@powroupi It is possible to switch functions by benchmark results. 😰 |
Oh, perhaps the result will change depending on the percentage of SDEF Vertex/All Vertex. |
I added the function to select the fastest strategy by benchmark.
|
mmd_sdef benchmark: default 2.6922 vs bulk_update 3.1725 => use On my Windows 10 PC, the bulk update version (np version) is slower. |
An imported MMD model may have SDEF errors. Previously mmd_tools was only concerned about the import and export of SDEF data, without making any corrections. If an imported model has incorrect or incomplete C or R0 or R1 values, will these now be automatically re-calculated and corrected by the add-on of @nagadomi ? |
Comparing with MMD, I think this version is more correct. 😃 for bone0, bone1, sdef_data, vids in g_mmd_sdef_verts[obj.name].values():
mat0 = bone0.matrix * bone0.bone.matrix_local.inverted()
mat1 = bone1.matrix * bone1.bone.matrix_local.inverted()
rot0 = mat0.to_quaternion()
rot1 = mat1.to_quaternion()
if rot1.dot(rot0) < 0:
rot1 = -rot1
# s0, s1 = mat0.to_scale(), mat1.to_scale() # for scaling
for vid, w0, w1, pos_c, cr0, cr1 in sdef_data:
mat_rot = (rot0*w0 + rot1*w1).normalized().to_matrix()
# s = s0*w0 + s1*w1
# mat_rot *= Matrix([[s[0],0,0], [0,s[1],0], [0,0,s[2]]])
shapekey_data[vid].co = mat_rot * pos_c + mat0 * cr0 * w0 + mat1 * cr1 * w1 Sample file: sdef_test.pmx.zip |
@powroupi |
Thank you all for the amazing work! Sadly I couldn't contribute, since this is above my current level of python skills. Using Aria as a benchmark, I think the speed is good enough when rendering, although I would disable SDEF for posing. Or the parts that do the heavy math could be compiled using cython. That would require separate compiled versions for Linux/Windows/MacOS, so it should probably be an optional install that can replace the uncompiled SDEF python file for people who want more performance. Would this be feasible? |
Just minor issues for future improvement, currently your code is no problem, you can sent PR to me 😄
key_blocks = tuple(k for k in obj.data.shape_keys.key_blocks[1:] if not k.mute and k.name != "mmd_sdef_skinning")
for bone0, bone1,...
for vid, w0, w1,...
offset = sum((k.value*(k.data[vid].co-k.relative_key.data[vid].co) for k in key_blocks), Vector())
shapekey_data[vid].co = rot*(pos_c+offset) + mat0*cr0*w0 + mat1*cr1*w1 - offset
|
Okay, I've updated SDEF feature, now we can bind/unbind selected (mesh) objects (select MMD model root object to bind/unbind all meshes of the MMD model), we can also mute/unmute SDEF shape key or toggle [SDEF] in [MMD Display] panel. 😄 |
I am not familiar with weight painting of SDEF. |
According to information in a tutorial, there is a Metasequoia plug-in called Keynote, which is used to add SDEF to PMX models. Are Metasequoia plug-ins programmed in Python? If Metasequoia plug-ins use Python, then, translating code from Keynote to Blender Python should work. |
@nagadomi I read the linked blog entry via Google translate. My understanding of MMD's anchor setting is that it is used for altering weights, not SDEF C/R0/R1 values. It's used for editing of BDEF as well. I've never used this functionality because it's not very intuitive, and because I can use Blender to achieve the same sorts of things that PMXE's anchors are used for-- things like locking vertex groups and renormalizing other weights around them. I don't believe the author is using anchors to change SDEF centers, but to change weights to make the model work with the SDEF centers calculated by PMXE. Google translate gives me, "If this remains the default value (during simple SDEF conversion) The inside is dented and the outside bulges." which isn't quite accurate. Sometimes, default values cause undesirable deformation; usually, they do not. PMXE's transformation from BDEF->SDEF calculates SDEF values automatically on the basis of the locations of the bones involved. When it's not, I move my bones and do it over again. After calculating SDEF, I can move my bones wherever I want them to go without changing the SDEF values. Part of that may be the fact that I use SDEF judiciously, on only parts of my models, and make sure that I have BDEF1 loops at every SDEF/BDEF border. As a recent example of how PMXE's automatically calculated centers can go wrong, I recently had a model with a (near) T-pose. Humerus twist bone was coincident with the humerus bone (only the axis matters for transformation of an axis-limited bone like this, distance along axis is unimportant.) Automatically calculated SDEF values for the arm twist bones were beautiful. But upon transformation to an A pose (and SDEF recalculation) for better compatibility with existing motions, arm twist bones deformed very poorly. I moved my humerus twist bones to a more standard position, closer to the elbow along the same axis, recalculated SDEF, and the problem went away. (I could have moved them back after calculation, but again, as twist bones, their position along the axis doesn't actually matter.) |
@nathanvasil Actually the overlap of the anchor shapes changes the volume of the deformation a bit. I don't know which of the values it changes though. I sometimes used it to fix knees. It seems that overlapping two box anchors for thigh+shin, turning the backside really flat and the front really tall, so it looks like a pizza slice from the side, with the shin bone handle (circle) in the narrow end and the tall end fully enveloping the knee, gives pointier knees than the default BDEF->SDEF conversion. So there is definitely something that works different when using this method. |
I would be interested in learning more about PMXE anchors, @Takuyax, and I would appreciate anything you have to share on the subject. |
suwatoh's pmx exporter for blender(exp_pmx-1.5.2.zip) supports automatically setting of SDEF parameters from the distribution of vertex positions and the bones position. It is licensed under NSYL (it is similar to CC0). sdef_key = (bi1, bi2)
sdef_data = sdef_map.get(sdef_key)
if sdef_data is None:
pos1 = bone_specs[bi2]["position"]
pos2 = bone_specs[bi1]["position"]
vec1 = (pos2 - pos1).normalized()
vec2 = co - pos1
dist_pos1_pi = vec1.dot(vec2)
sdef_c = pos1 + (vec1 * dist_pos1_pi)
sdef_map[sdef_key] = [[index], pos1, vec1, dist_pos1_pi, dist_pos1_pi]
else:
sdef_data[0].append(index)
pos1 = sdef_data[1]
vec1 = sdef_data[2]
vec2 = co - pos1
dist_pos1_pi = vec1.dot(vec2)
sdef_c = pos1 + (vec1 * dist_pos1_pi)
sdef_data[3] = max(sdef_data[3], dist_pos1_pi)
sdef_data[4] = min(sdef_data[4], dist_pos1_pi)
vert_spec["deform"] = 3 # SDEF
vert_spec["bone1"] = bi1
vert_spec["bone2"] = bi2
vert_spec["weight1"] = weight1 / weight_sum
vert_spec["weight2"] = sdef_c # C
vert_spec["weight3"] = ZERO_VECTOR # R0
vert_spec["weight4"] = ZERO_VECTOR # R1 R0, R1 will be recalculated later. for sdef_data in sdef_map.values():
for index in sdef_data[0]:
vert_spec = vertices[index]
if vert_spec["deform"] != 3:
continue
pos = sdef_data[1]
nvec = sdef_data[2]
vert_spec["weight3"] = pos + (nvec * sdef_data[3]) # R0
vert_spec["weight4"] = pos + (nvec * sdef_data[4]) # R1 |
@nathanvasil Okay, I did some research about what the anchor editor actually does. The way I use it is probably a bit unusual, because I only use it to add SDEF to knees and elbows if models don't have it yet. So I shape the anchors to form a wedge that only affects the area I want to change. They cover almost exactly the same space. If there is space that only one of them covers, it turns everything in that non-overlapping space into BDEF1 weights. |
Thanks @Takuyax, I'll check that out! Don't have a bowlroll account which I'd need to download that model, but maybe I'll register, or just try playing with anchors and see if I can figure it out (I like using simple test cases, cubes and such, to figure out how tools like that work.) |
You can log into bowlroll.net with a twitter, facebook, or github account. |
Hi!
This issue is mainly intended as a discussion that might lead to some enhancements in mmd tools, or to myself understanding how to do this manually.
My main usage scenario is to use MMD tools as an importer to render models directly in Blender. Models get imported, converted to quads, and I do a bunch of blender-specific manual adjustments (separate mesh objects, creases, lattice deformed spherical eyes, etc.)
To get nice pointy elbows and knees, I manually add a second armature modifier, set it to “Preserve Volume” (which activates quaternion interpolation) and “Multi Modifier”, then add a vertex group “volume”, which controls the influence of the quaternion armature modifier. I then pose the model and weight paint the “volume” vertex group until I get a nice shape.
This manual approach works great, however every time I see that a model has SDEF shape keys, I wonder if it wouldn’t be possible to convert this to something that blender could use, even if that would be a simplified approximation.
For starters, is there any good documentation about how the c, r0 and r1 coordinates work? I suppose c stands for center and r for radius, so it would seem like it’s possible to calculate some basic per-vertex volume data from these. Why are are two radiuses though?
The text was updated successfully, but these errors were encountered: