Creating entities doesn't let me specify "just a Kind"

[jgeewax]

To create a new Product entity, I have to do the following

from gcloud import datastore
product = datastore.Entity(key=datastore.Key('Product'))

This seems unnecessarily long to me. How about

from gcloud import datastore
product = datastore.Entity('Product')

or

from gcloud import datastore
product = datastore.Entity.from_kind('Product')

or

from gcloud import datastore
product = datastore.Kind('Product')

[dhermes]

I am very opposed to adding Kind. This is not an ORM.

We had previously discussing allow *args, **kwargs into the Entity() constructor and letting the *args go towards creating a key. The conversation got held up when we tried to send properties in through **kwargs since there could be name collisions with kwargs we actually needed to use (key and exclude_from_indexes).

I am mildly opposed to hiding Key in the constructor because user's lose track of the concept. We can counteract this by having docs snippets like

e = Entity('Foo')
datastore.put([e])
datastore.delete([e.key])

---

Suggested middle-ground: Make key a positional only argument. If len(args) == 1 we check if it is of type Key (I verified this wouldn't cause an import cycle) and if not we construct the key from the args.

The ndb base Model pulls this off by calling

    get_arg = self.__get_arg
    key = get_arg(kwds, 'key')
    id = get_arg(kwds, 'id')
    app = get_arg(kwds, 'app')
    namespace = get_arg(kwds, 'namespace')
    parent = get_arg(kwds, 'parent')
    projection = get_arg(kwds, 'projection')

and then doing a ton of validation logic right in the constructor.

The Model.__get_arg class method checks first for _app (or _id, _key, etc.) and then if _app is not in kwargs it checks if app is a property or model key on the model definition. If app is not a property, then it can be popped from the kwargs.

[dhermes]

@tseaver Do you have opinions on consequences of enabling:

product = datastore.Entity('Product')

[tseaver]

I don't like it -- users of the API really need to understand keys vs. entities to make appropriate use of it. I'm not a big fan of "saving typing" when that leads to a "hockey-stick" learning curve one week into using the library.

I'm actually not a big fan of even the datastore.Key and datastore.Entity spellings: saving the extra imports:

from gcloud.datastore.key import Key
from gcloud.datastore.entity import Entity

is a false win compared to how the majority of the code would look afterwards:

product = Entity(Key('Product'))

[jgeewax]

I think the important thing to note here is that I'm I want to create an entity of a certain kind, and I don't want to think about what the key is -- I want the library to come up with that.

So saying product = Entity(Key('Product')) introduces a new concept (a Key) that I don't want to think about yet.

Re the false savings... if we have a way to quickly create entities without thinking about a Key, this would become:

from gcloud import datastore

person1 = datastore.Entity.from_kind('Person')
datastore.put(person1)

Which seems like a great win.

[dhermes]

I am also against from_kind.

• A fast/easy onboarding experience is nice, but it's very difficult to use the datastore API without knowing what a key is. It's not our concept to hide, it's a core part of the API.
• How can a user know what a kind is without knowing what a key is?
• Adding a from_kind method adds another way to do something that is already possible
• The suggested from_kind(...) is the same (same-ish) number of keystrokes as using Key(...).
• We can do creative things to make the constructor of Entity allow key construction. (See my "suggested middle-ground" above.)

[jgeewax]

I am fine with the middle ground (Entity('Foo')).

it's very difficult to use the datastore API without knowing what a key is. It's not our concept to hide, it's a core part of the API.

I'll agree that it's a core part of the API, however the fact that it composes a Kind and a Name/ID is complicated and I'm happy to hide that combination when creating new entities.

If I'm setting the kind of the Entity, I don't think "I'll set the Kind on a Key, and the Key on the Entity". I think "I'll create a new entity of a Kind x".

I guess that I'm saying is: I am perfectly comfortable with people constructing new entities without thinking about the incompleteness of the Key and the fact that an Entity always has a Key even when it's not created yet.

So Entity.from_kind('Foo') and Entity('Foo') are both acceptable to me.

And on that note: An entity is basically a Key + a dictionary, right? So why not just pass along all the constructor arguments? What's the logic there?

[dhermes]

RE: "why not just pass along all the constructor arguments?"

This has been discussed before. The current signature is:

def __init__(self, key=None, exclude_from_indexes=()):

so to do this we'd end up swapping

e = Entity(key, exclude_list)
e.update(prop_dict)

for

e = Entity(key, **prop_dict)
e.exclude_indexes(exclude_list)

It's not something we were ever strongly for or against, but the fact that we don't get any simplification (just shifting responsibility) made it so that no one pushed too hard on it.

[jgeewax]

I was suggesting that we pass along the Entity constructor's arguments to the Key constructor, making the "Entity" itself basically a Key + a dictionary.

def __init__(self, *key_args, **key_kwargs):
  self.key = Key(*key_args, **key_kwargs)

So that I could write

person = Entity('Person')
entity = Entity('Foo', namespace='namespace', parent=person.key)
entity.update({'foo': 'val'})
entity.exclude_indexes(...)
datastore.put([person, entity])

Just seems off that the Entity class needs a lot of stuff, but it's really all needed ultimately by the Key.

[dhermes]

The only thing "ultimately" needed by the key is the key arguments. The rest is unique to the entity (e.g. the indexed / excluded fields).

The patterns above are just sugar that ends up hiding the concept of the Key all together. What is a parent if there is no such thing as a key?

Anyhow, I think we need to fight the urge to be an ORM since ndb is well on its way.

[jgeewax]

The indexes piece will be changing, at which point the only argument for an entity will be a Key... Further, the index specification is rarely used, and not very well defined in its current form (hence why we'll be changing it).

IMO, an Entity is just the Key with data, so I don't think this pushes us in the direction of an ORM, but just makes it more convenient to do the common cases.

What is a parent if there is no such thing as a key?

A parent can be another Entity (that's how a lot of people think of it anyway -- few people realize you can have many keys all with the same "parent" where the parent entity actually doesn't exist).

Breaking this down into the kind of stuff I want to do:

1. Create a new entity of a specific kind and an auto-assigned ID
2. Create a new entity with a specific kind, auto-assigned ID, and specific parent
3. Create a new entity with a specific key
4. Create a new entity with a specific key and specific parent
5. Look-up an entity with a specific key and do something with it (save it? delete it?)
6. Run a query which brings back lots of entities and do something with those (look at them? update them? delete them?)

The code I'd want to write for each would be:

from gcloud import datastore

# Create a new entity, auto-assigned ID.
person = datastore.Entity('Person')
# Note that I don't want to do: `person = datastore.Entity(datastore.Key('Person'))`
# because it feels a bit backwards to create a Key that isn't really a Key -- it's a partial key... 
person['name'] = 'Danny'
datastore.put(person)

# Create a new entity with a specific kind, auto-assigned ID, and specific parent
parent = datastore.Entity('Person')
parent['name'] = 'Parent'
datastore.put(parent)
child = datastore.Entity('Person', parent=parent) # Note, not parent.key
child['name'] = 'Child'
datastore.put(child)

# Create a new entity with a specific key
person = datastore.Entity('Person', 'dhermes')
# Note this is also OK IMO: `person = datastore.Entity(key=datastore.Key('Person', 'dhermes'))`
person['name'] = 'Danny'
datastore.put(person)

# Create a new entity with a specific key and specific parent
parent = datastore.Entity('Person', 'parent')
parent['name'] = 'Parent'
child = datastore.Entity('Person', 'child', parent=parent)
child['name'] = 'Child'
datastore.put([parent, child])

# Look-up an entity
person = datastore.get(datastore.Key('Person', 1234))
person2 = datastore.get(datastore.Key('Person', 'Danny'))

# Run a query
query = datastore.Query('Person').add_filter('name', '=', 'Danny')
for person in query:
  person['active'] = True
  datastore.put(person)

[dhermes]

TLDR; @jgeewax The changes you suggest in your last two comments just make Entity effectively a subclass of Key that also holds key->value data and index metadata.

The core thing I'm uncomfortable with is blurring the actual API concepts with our own. It makes the process of understanding the datastore API service difficult for our users.

---

RE: "The indexes piece will be changing." Is there some document to see? It's hard for me to imagine how our code should look based on an amorphous future.

RE: "Further, the index specification is rarely used" Evidence? It's not even possible to insert an indexed string > 500 chars or an indexed Entity in any form. So of course it's used plenty, because people store strings longer than 500 chars and store sub-entities. In addition people turn off indexing to avoid both cost ($) and slowdown (time) in writes.

RE: "an Entity is just the Key with data" I totally agree, but there are two types of data associated to the Key, the key->value pairs and the index metadata. This was part of the many discussions with Eli. (The index stuff may change, based on what you said, but I can't see the future from the present.)

RE: "A parent can be another Entity (that's how a lot of people think of it anyway..." Which people? That's a sign of mis-education of users. Not something we should help propagate.

RE: "it feels a bit backwards to create a Key that isn't really a Key -- it's a partial key... " This should be addressed with the API devs, not with the library that implements it. A Key can be incomplete.

[jgeewax]

RE: "The indexes piece will be changing." Is there some document to see? It's hard for me to imagine how our code should look based on an amorphous future.

No doc yet, but it will not be managed by code. Indexes will be managed in a way much more similar to how you handle indexes on a SQL instance (through some separate administrative API).

RE: "Further, the index specification is rarely used" Evidence? It's not even possible to insert an indexed string > 500 chars or an indexed Entity in any form. So of course it's used plenty, because people store strings longer than 500 chars and store sub-entities. In addition people turn off indexing to avoid both cost ($) and slowdown (time) in writes.

I'll pull some data. Regardless, with the change above, this will be less of an issue. The fact that we have to set which properties to leave out of indexes in code is a bummer.

RE: "an Entity is just the Key with data" I totally agree, but there are two types of data associated to the Key, the key->value pairs and the index metadata. This was part of the many discussions with Eli. (The index stuff may change, based on what you said, but I can't see the future from the present.)

I see the index metadata associated with the Kind rather than the Entity. And we're changing Datastore to reflect that. If that were to go away, I think we're in agreement that an Entity is simply a Key with key->value pairs. Index metadata lives on the Kind and should be managed via an administrative API, not when you persist a single entity.

RE: "A parent can be another Entity (that's how a lot of people think of it anyway..." Which people? That's a sign of mis-education of users. Not something we should help propagate.

"An entity can also have an optional parent entity." See https://cloud.google.com/appengine/docs/python/datastore/modelclass

In App Engine, we pitched Entity Groups as Entities that have the same Parent entity. It was never clear from our docs that the Parent Entity didn't need to exist, and that Keys could have hierarchy regardless of whether the Entity existed or not.

Another point of reference for that: http://stackoverflow.com/questions/12847062/if-an-entitys-ancestor-does-not-exist-will-it-be-created-java-api

RE: "it feels a bit backwards to create a Key that isn't really a Key -- it's a partial key... " This should be addressed with the API devs, not with the library that implements it. A Key can be incomplete.

Let's say you're right, our job is still usability. person = datastore.Entity(datastore.Key('Person')) is overly verbose IMO. person = datastore.Entity('Person') is what I'd expect to type to get back an Entity of type 'Person' that I can then persist and get an automatically generated ID back.

[dhermes]

This all really hinges on the concept of Kind as an API concept rather than a library concept (with ndb and db and a few others).

It all seems to unfold nicely in the future you've described, though I am very wary of doing something before it exists.

As for verbosity,

person = Entity(key=Key('Person'))
# is longer than
person = Entity('Person')

but again the extra 8 characters is worth teaching the user what they are making. The API concepts of Entity and Key are distinct. (This falls into the same boat: if a Kind exists they make sense.)