mongodb - MongoEngine - How to deference a List Field efficiently when converting to json -

class parent(document):     name = stringfield()     children = listfield(referencefield('child'))  class child(document):     name = stringfield()     parents = listfield(referencefield(parent))  @app.route('/home/') def home():     parents = parent.objects.all()     return render_template('home.html', items=parents) 

i have 2 collections similar above, maintain many many relationship.

in template angular, i'm setting javascript variable list of parents so:

$scope.items = {{ parents|tojson }}; 

this results in array of parents who'se chilren array of object ids (references), opposed dereferenced child objects:

$scope.items = [{'$oid': '123', 'name': 'foo', 'children': [{'$oid': '456'}]}]; 

i want angular object contain of dereferenced children. there efficient way this?

so far, approach works me, @ o(n^3). i've minimized list comprehensions clarity. multiple obj['_id'] = {'$oid': str(obj['_id']} necessary convert objectid can serialized json.

@app.route('/home/') def home():     parents = parent.objects.all()     temps = []     parent in parents:         p = parent.to_mongo()         # @ point, children of parent , p references         p['_id'] = {'$oid': str(p['_id'])         temp_children = []         child in parent.children:             # child dereferenced             c = child.to_mongo()             c['_id'] = {$oid': str(c['_id'])}             # children have links parent. keep these references.             c['parents'] = [{'oid': str(parent_ref)} parent_ref in c['parents']]             temp_children.append(c)          p['children'] = temp_children         temps.append(parent.to_mongo())      return render_template('home.html', items=temps)             

the following not work result in non-dereferenced children:

json.loads(json.dumps(accounts)) 

because storing children references going have go server dereference them when using queryset.all method above. folks @ mongodb know big performance problem when using drivers pymongo have aggregation framework allow dereferencing on server.

the documentation using mongoengine pretty poor @ unit tests in mongoengine source fill in blanks.

with this answer , if using mongodb 3.2 or later can achieve looking follows:

import mongoengine db bson.json_util import dumps  class parent(db.document):     name = db.stringfield()     children = db.listfield(db.referencefield('child'))   class child(db.document):     name = db.stringfield()     parents = db.listfield(db.referencefield(parent))   pipeline = [{"$unwind": "$children"},             {"$lookup":                  {"from": "child",                   "localfield": "children",                   "foreignfield": "_id",                   "as": "children"                   }              },             {"$group": {                 "_id": "$_id",                 "name": {"$first": "$name"},                 "children": {"$push": "$children"}             }             }             ]   @app.route('/home/') def home():     parents = []     p in parent.objects.aggregate(*pipeline):         parents.append(p)     items= dumps(parents)     return render_template('home.html', items=items) 

then in home.html need:

$scope.items = {{ items }}; 

the basic steps in pipeline here are:

1. unwind children: make separate document every child element in children array
2. lookup children: go child collection , lookup based on _id , store result in children field on each document. replacing objectid matching document.
3. group results: _id , include name based on first item in grouping , push children fields field named children

$lookup available mongodb 3.2 , if need run version of mongodb you'll have no choice make multiple queries. also, $lookup won't work on sharded collections.