json2pattern fails with some aggregate pipelines

[divyanshrai]

I'm trying to make a tool for 4.4 logs, I'm using some parts of code from this repository during which I noticed this bug.

The json2pattern function which converts JSON format to a query pattern fails under some aggregate pipelines. This function is present at mtools/util/pattern.py
I'm going to take two examples here.
Example 1 -
[{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$project": {"id": 1, "subject": 1, "status": 1, "created_at": 1, "requester": 1, "assignee": 1, "jira": 1, "jira_status": 1, "priority": 1, "type": 1, "updated_by": 1, "updated_at": 1, "unread_comment_count": 1, "last_comment_unread_at": 1, "custom_fields": 1, "_id": 0, "form_id": 1, "brand_id": 1, "metric.resolution_time": 1}}, {"$sort": {"created_at": -1}}, {"$skip": 0}, {"$limit": 25}]

Example 2-
[{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$group": {"_id": 1, "n": {"$sum": 1}}}]

Expected behavior

Of course, I expected it to return something like this -

Example 1 -

[{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["1"]}}]}, {"$and": [{"brand_id": {"$in": ["1"]}}, {"status": {"$nin": ["1"]}}, {"form_id": {"$nin": ["1"]}}, {"custom_fields": {"$elemMatch": {"id": "1", "value": {"$in": ["1"]}}}}]}]}}, {"$group": {"_id": 1 , "n": {"$sum": 1 }}}]

Example 2-

[{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["1"]}}]}, {"$and": [{"brand_id": {"$in": ["1"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["1"]}}, {"custom_fields": {"$elemMatch": {"id": "1", "value": {"$in": ["1"]}}}}]}]}}, {"$project": {"id": 1 , "subject": 1 , "status": 1 , "created_at": 1 , "requester": 1 , "assignee": 1 , "jira": 1 , "jira_status": 1 , "priority": 1 , "type": 1 , "updated_by": 1 , "updated_at": 1 , "unread_comment_count": 1 , "last_comment_unread_at": 1 , "custom_fields": 1 , "_id": 1 , "form_id": 1 , "brand_id": 1 , "metric.resolution_time": 1 }}, {"$sort": {"created_at": 1 }}, {"$skip": 1 }, {"$limit": 1 }]

Actual/current behavior

I'm going to paste the output which I got after using "debug=True"
Example 1-

=======================
 [{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$project": {"id": 1, "subject": 1, "status": 1, "created_at": 1, "requester": 1, "assignee": 1, "jira": 1, "jira_status": 1, "priority": 1, "type": 1, "updated_by": 1, "updated_at": 1, "unread_comment_count": 1, "last_comment_unread_at": 1, "custom_fields": 1, "_id": 0, "form_id": 1, "brand_id": 1, "metric.resolution_time": 1}}, {"$sort": {"created_at": -1}}, {"$skip": 0}, {"$limit": 25}]
[{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$project": {"id": 1, "subject": 1, "status": 1, "created_at": 1, "requester": 1, "assignee": 1, "jira": 1, "jira_status": 1, "priority": 1, "type": 1, "updated_by": 1, "updated_at": 1, "unread_comment_count": 1, "last_comment_unread_at": 1, "custom_fields": 1, "_id": 0, "form_id": 1, "brand_id": 1, "metric.resolution_time": 1}}, {"$sort": {"created_at": -1}}, {"$skip": 0}, {"$limit": 25}]
[{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$project": {"id": 1, "subject": 1, "status": 1, "created_at": 1, "requester": 1, "assignee": 1, "jira": 1, "jira_status": 1, "priority": 1, "type": 1, "updated_by": 1, "updated_at": 1, "unread_comment_count": 1, "last_comment_unread_at": 1, "custom_fields": 1, "_id": 0, "form_id": 1, "brand_id": 1, "metric.resolution_time": 1}}, {"$sort": {"created_at": -1}}, {"$skip": 0}, {"$limit": 25}]
[{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$project": {"id": 1 , "subject": 1 , "status": 1 , "created_at": 1 , "requester": 1 , "assignee": 1 , "jira": 1 , "jira_status": 1 , "priority": 1 , "type": 1 , "updated_by": 1 , "updated_at": 1 , "unread_comment_count": 1 , "last_comment_unread_at": 1 , "custom_fields": 1 , "_id": 1 , "form_id": 1 , "brand_id": 1 , "metric.resolution_time": 1 }}, {"$sort": {"created_at": 1 }}, {"$skip": 1 }, {"$limit": 1 }]
[{"$match": {"$and": [{"$and": [{"brand_id": {"$in": [ 1 ]}}}}]}]}}, {"$project": {"id": 1 , "subject": 1 , "status": 1 , "created_at": 1 , "requester": 1 , "assignee": 1 , "jira": 1 , "jira_status": 1 , "priority": 1 , "type": 1 , "updated_by": 1 , "updated_at": 1 , "unread_comment_count": 1 , "last_comment_unread_at": 1 , "custom_fields": 1 , "_id": 1 , "form_id": 1 , "brand_id": 1 , "metric.resolution_time": 1 }}, {"$sort": {"created_at": 1 }}, {"$skip": 1 }, {"$limit": 1 }]
json2pattern():json.loads Exception:
  Error: <class 'json.decoder.JSONDecodeError'> : Expecting ',' delimiter: line 1 column 61 (char 60)
  saved_s: ([{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$project": {"id": 1, "subject": 1, "status": 1, "created_at": 1, "requester": 1, "assignee": 1, "jira": 1, "jira_status": 1, "priority": 1, "type": 1, "updated_by": 1, "updated_at": 1, "unread_comment_count": 1, "last_comment_unread_at": 1, "custom_fields": 1, "_id": 0, "form_id": 1, "brand_id": 1, "metric.resolution_time": 1}}, {"$sort": {"created_at": -1}}, {"$skip": 0}, {"$limit": 25}])
  s: ([{"$match": {"$and": [{"$and": [{"brand_id": {"$in": [ 1 ]}}}}]}]}}, {"$project": {"id": 1 , "subject": 1 , "status": 1 , "created_at": 1 , "requester": 1 , "assignee": 1 , "jira": 1 , "jira_status": 1 , "priority": 1 , "type": 1 , "updated_by": 1 , "updated_at": 1 , "unread_comment_count": 1 , "last_comment_unread_at": 1 , "custom_fields": 1 , "_id": 1 , "form_id": 1 , "brand_id": 1 , "metric.resolution_time": 1 }}, {"$sort": {"created_at": 1 }}, {"$skip": 1 }, {"$limit": 1 }])

Example 2 -

[{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$group": {"_id": 1, "n": {"$sum": 1}}}]
[{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$group": {"_id": 1, "n": {"$sum": 1}}}]
[{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$group": {"_id": 1, "n": {"$sum": 1}}}]
[{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$group": {"_id": 1 , "n": {"$sum": 1 }}}]
[{"$match": {"$and": [{"$and": [{"brand_id": {"$in": [ 1 ]}}}}]}]}}, {"$group": {"_id": 1 , "n": {"$sum": 1 }}}]
json2pattern():json.loads Exception:
  Error: <class 'json.decoder.JSONDecodeError'> : Expecting ',' delimiter: line 1 column 61 (char 60)
  saved_s: ([{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$group": {"_id": 1, "n": {"$sum": 1}}}])
  s: ([{"$match": {"$and": [{"$and": [{"brand_id": {"$in": [ 1 ]}}}}]}]}}, {"$group": {"_id": 1 , "n": {"$sum": 1 }}}])

Steps to reproduce the actual/current behavior

I can paste the loglines I've used -

1. 2022-02-28 01:53:46,297 pattern not found for logline - {"t": {"$date": "2022-02-22T23:17:30.970+00:00"}, "s": "I", "c": "COMMAND", "id": 51803, "ctx": "conn258", "msg": "Slow query", "attr": {"type": "command", "ns": "support_portal.tickets", "appName": "go_support_portal", "command": {"aggregate": "tickets", "allowDiskUse": true, "collation": {"locale": "en"}, "maxTimeMS": 120000, "pipeline": [{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$project": {"id": 1, "subject": 1, "status": 1, "created_at": 1, "requester": 1, "assignee": 1, "jira": 1, "jira_status": 1, "priority": 1, "type": 1, "updated_by": 1, "updated_at": 1, "unread_comment_count": 1, "last_comment_unread_at": 1, "custom_fields": 1, "_id": 0, "form_id": 1, "brand_id": 1, "metric.resolution_time": 1}}, {"$sort": {"created_at": -1}}, {"$skip": 0}, {"$limit": 25}], "cursor": {}, "lsid": {"id": {"$uuid": "68a15786-4973-4570-a17a-309ac66b6a1d"}}, "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1645571839, "i": 1}}, "signature": {"hash": {"$binary": {"base64": "5ffDD0yegZFTitrqubdlY6eygTE=", "subType": "0"}}, "keyId": 7009862960419962885}}, "$db": "support_portal", "$readPreference": {"mode": "primary"}}, "planSummary": "IXSCAN { brand_id: 1, custom_fields.id: 1, custom_fields.value: 1, created_at: -1 }", "keysExamined": 8506, "docsExamined": 8506, "hasSortStage": true, "cursorExhausted": true, "numYields": 11, "nreturned": 25, "queryHash": "EB85480F", "planCacheKey": "2620CEE1", "reslen": 127583, "locks": {"ReplicationStateTransition": {"acquireCount": {"w": 13}}, "Global": {"acquireCount": {"r": 13}}, "Database": {"acquireCount": {"r": 13}}, "Collection": {"acquireCount": {"r": 13}}, "Mutex": {"acquireCount": {"r": 2}}}, "storage": {"data": {"bytesRead": 59753313, "timeReadingMicros": 43316}}, "protocol": "op_msg", "durationMillis": 202}}

2. 2022-02-28 01:53:46,297 pattern not found for logline - {"t": {"$date": "2022-02-22T23:17:30.969+00:00"}, "s": "I", "c": "COMMAND", "id": 51803, "ctx": "conn250", "msg": "Slow query", "attr": {"type": "command", "ns": "support_portal.tickets", "appName": "go_support_portal", "command": {"aggregate": "tickets", "collation": {"locale": "en"}, "maxTimeMS": 120000, "pipeline": [{"$match": {"$and": [{"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3", "ckq2dv5s40003w7d2250o10uj"]}}]}, {"$and": [{"brand_id": {"$in": ["ckpwemryo0000w7d26aq4e0m3"]}}, {"status": {"$nin": ["solved", "closed"]}}, {"form_id": {"$nin": ["ckj1doiw402jssav2v1h52v8o", "ckj1doiw402jrsav2oao00uyn"]}}, {"custom_fields": {"$elemMatch": {"id": "ckjfymwqt06inylv2ozjr8t27", "value": {"$in": ["accounting_module"]}}}}]}]}}, {"$group": {"_id": 1, "n": {"$sum": 1}}}], "cursor": {}, "lsid": {"id": {"$uuid": "cc1783ad-adb1-4f63-ba81-62f8e475602a"}}, "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1645571839, "i": 1}}, "signature": {"hash": {"$binary": {"base64": "5ffDD0yegZFTitrqubdlY6eygTE=", "subType": "0"}}, "keyId": 7009862960419962885}}, "$db": "support_portal", "$readPreference": {"mode": "primary"}}, "planSummary": "IXSCAN { brand_id: 1, custom_fields.id: 1, custom_fields.value: 1, created_at: -1 }", "keysExamined": 8506, "docsExamined": 8506, "cursorExhausted": true, "numYields": 11, "nreturned": 1, "queryHash": "95329488", "planCacheKey": "198498DD", "reslen": 260, "locks": {"ReplicationStateTransition": {"acquireCount": {"w": 13}}, "Global": {"acquireCount": {"r": 13}}, "Database": {"acquireCount": {"r": 13}}, "Collection": {"acquireCount": {"r": 13}}, "Mutex": {"acquireCount": {"r": 2}}}, "storage": {"data": {"bytesRead": 22056185, "timeReadingMicros": 19168}}, "protocol": "op_msg", "durationMillis": 202}}

Environment

Since I'm using only one file of the repository, I can't mention any environment details.

Software	Version
mtools
Python	3.8.2
MongoDB server	4.4
Operating system	MacOS