Support synthetic source for aggregate_metric_double when ignore_malformed is used #108746
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
|
Hi @lkts, I've created a changelog YAML for you. |
lkts
commented
May 16, 2024
| * Build a {@link BytesRef} wrapping a byte array containing an encoded form | ||
| * of the passed XContentBuilder contents. | ||
| */ | ||
| public static BytesRef encodeXContentBuilder(XContentBuilder builder) throws IOException { |
kkrik-es
reviewed
May 17, 2024
...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java
Outdated
Show resolved
Hide resolved
kkrik-es
reviewed
May 17, 2024
...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java
Outdated
Show resolved
Hide resolved
kkrik-es
reviewed
May 17, 2024
| @@ -506,6 +572,9 @@ private Map<String, Object> randomAggregateMetric() { | |||
| private void mapping(XContentBuilder b) throws IOException { | |||
| String[] metrics = storedMetrics.stream().map(Metric::toString).toArray(String[]::new); | |||
| b.field("type", CONTENT_TYPE).array(METRICS_FIELD, metrics).field(DEFAULT_METRIC, metrics[0]); | |||
| if (malformedExample) { | |||
| b.field(IGNORE_MALFORMED, true); | |||
| } | |||
| } | |||
|
|
|||
There was a problem hiding this comment.
Can we add some more elaborate use cases, here or in yaml? We want to check for:
- Nested subfields within aggregate_metric_double
- Partial aggregate_metric_double (e.g. one metric ok, one invalid, two ok)
- Nested malformed fields, with other fields at the same level
There was a problem hiding this comment.
We do cover partials in malformedValue. Nested is a good suggestion.
kkrik-es
reviewed
May 17, 2024
server/src/main/java/org/elasticsearch/common/xcontent/XContentHelper.java
Show resolved
Hide resolved
kkrik-es
reviewed
May 17, 2024
...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java
Outdated
Show resolved
Hide resolved
kkrik-es
reviewed
May 17, 2024
...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java
Show resolved
Hide resolved
lkts
changed the title
|
@elasticmachine update branch |
kkrik-es
reviewed
May 22, 2024
| metric: | ||
| min: 18.2 | ||
| max: 100 | ||
| value_count: 50 |
There was a problem hiding this comment.
Nit: move value_count to the end, to make sure it's still tracked.
kkrik-es
approved these changes
May 22, 2024
martijnvg
reviewed
May 22, 2024
...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java
Show resolved
Hide resolved
| // We don't use DrainingXContentParser since we don't want to go beyond current field | ||
| malformedContentForSyntheticSource.copyCurrentStructure(context.parser()); | ||
| } | ||
| ; |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds synthetic source support for
aggregate_metric_doublefield whenignore_malformedis used.This PR introduces a pattern that will be reused in
ignore_malformedsupport in synthetic source for other (complex object-like) fields. The pattern is to create a "shadow"XContentBuilderthat replicates all successfully parsed fields and values. In case of malformed data, everything remaining in the parser (inside the field) is copied over to the builder. As a result we get both successfully parsed pieces, malformed piece, and skipped pieces which is a full representation of user input and can go to synthetic source.Contributes to #90007.