Schema transformation process should be able to preserve original attributes

[lmolkova]

During schema transformation, we remove one attribute and create a new one.

Assuming users have built some tooling based on old attributes, such tooling will be broken by applying schema transformation.

If users apply schema transformation explicitly themselves, it's fine, but for collector-distro or within vendor-specific pipelines this behavior would be problematic and breaking. Even if a vendor makes UI changes that hide renames, custom queries could still be broken.

Even when the transformation is enabled explicitly by users, they should have a choice to keep the original attributes to have time to gradually update tooling.

Schema transformation should support a mode when original data is preserved and this mode should be on by default.

[dmitryax]

If we keep the old values during the schema transformation, we alleviate the transition for users only when the schema transform processor is introduced. However, once any of the libraries are updated to emit a newer version or the transform processor is updated to emit a different version, the dashboards will no longer function properly anyway.

In my opinion, the schema transformation should only be employed once the backend and dashboards are prepared to consistently handle a specific schema version. Otherwise, the purpose of the transformation becomes unclear and not particularly beneficial. Users can simply upgrade their instrumentation when necessary and avoid using any transformations.

[lmolkova]

This issue is related to edge-case handling, the essential question is if renames should be allowed at all in stable conventions - #3513

[austinlparker]

One thing in support of this, we've seen many customers have difficulties around migrating from experimental to stable HTTP semantic conventions specifically around this tooling gap. If the schema was able to assist in dual-write of old and new attributes during the transition, they'd be able to more comfortably upgrade without breaking existing alerts, dashboards, etc.