The following is the first two items in my json file
{
"ReferringUrl": "N",
"OpenAccess": "0",
"Properties": {
"ItmId": "1694738780"
}
}
{
"ReferringUrl": "L",
"OpenAccess": "1",
"Properties": {
"ItmId": "1347809133"
}
}
I want to count the number of items by each ItmId appeared in the json. For example, items that with "ItmId" 1694738780 appears 10 times and items with "ItmId" 1347809133 appears 14 times in my json file. Then return a json like this
{"ItemId": "1694738780",
"Count": 10
}
{"ItemId": "1347809133",
"Count": 14
}
I am using bash. And prefer do this totally by jq. But it's ok to use other method.
Thank you!!!
Here's one solution (assuming the input is a stream of valid JSON objects) and that you invoke jq with the -s option:
map({ItemId: .Properties.ItmId}) # extract the ItmID values
| group_by(.ItemId) # group by "ItemId"
| map({ItemId: .[0].ItemId, Count: length}) # store the counts
| .[] # convert to a stream
A slightly more memory-efficient approach would be to use inputs
if your jq has it; but in that case, use -n instead of -s, and replace the first line above by: [inputs | {ItemId: .Properties.ItmId} ]
The above solutions use the built-in group_by
, which is convenient but leads to easily-avoided inefficients. Using the following counter
makes it easy to write a very efficient solution:
def counter(stream):
reduce stream as $s ({}; .[$s|tostring] += 1);
Using the -n command-line option, and applied as follows:
counter(inputs | .Properties.ItmId)
this leads to a dictionary of counts:
{
"1694738780": 1,
"1347809133": 1
}
Such a dictionary is probably more useful than a stream of singleton objects as envisioned by the OP, but if such as stream is needed, one can modify the above as follows:
counter(inputs | .Properties.ItmId)
| to_entries[]
| {ItemId: (.key), Count: .value}
Using jq command
cat json.txt | jq '.Properties .ItmId' | sort | uniq -c | awk -F " " '{print "{""ItmId"":" $2 ",""count"":" $1"}"}'| jq .
Here's a super-efficient solution -- in particular, no sorting is required. The following implementation requires a version of jq with inputs
but it is easy to adapt the program to use earlier versions of jq. Please remember to use the -n command-line option if using the following:
# Count the occurrences of distinct values of (stream|tostring).
# To avoid unwanted collisions, or to recover the exact values,
# consider using tojson
def counter(stream):
reduce stream as $s ({}; .[$s|tostring] += 1);
counter(inputs | .Properties.ItmId)
| to_entries[]
| {ItemId: (.key), Count: .value}
Here is a variation using reduce, setpath and getpath to do the aggregation and to_entries to do the final formatting which assumes you run jq as
jq --slurp -f query.jq < data.json
where data.json contains your data and query.jq contains
map(.Properties.ItmId)
| reduce .[] as $i (
{}; setpath([$i]; getpath([$i]) + 1)
)
| to_entries | .[] | { "ItemId": .key, "Count": .value }