Fixed #373 -- Added CompositePrimaryKey. #18056

csirmazbendeguz · 2024-04-07T04:28:04Z

Trac ticket number

Branch description

This branch adds the CompositePrimaryKey field. If present, Django will create a composite primary key.

Proposal
Previous PR
Composite FK
Serial Fields

class Tenant(models.Model):
    pass


class User(models.Model):
    primary_key = models.CompositePrimaryKey("tenant_id", "id")
    tenant = models.ForeignKey(Tenant, on_delete=models.CASCADE)
    id = models.IntegerField()


class Comment(models.Model):
    primary_key = models.CompositePrimaryKey("tenant_id", "id")
    tenant = models.ForeignKey(Tenant, on_delete=models.CASCADE)
    id = models.IntegerField()
    user_id = models.IntegerField()
    user = models.ForeignObject(
        User,
        on_delete=models.CASCADE,
        from_fields=("tenant_id", "user_id"),
        to_fields=("tenant_id", "id"),
        related_name="+",
    )

Checklist

This PR targets the main branch.
The commit message is written in past tense, mentions the ticket number, and ends with a period.
I have checked the "Has patch" ticket flag in the Trac system.
I have added or updated relevant tests.
I have added or updated relevant docs, including release notes if applicable.
For UI changes, I have attached screenshots in both light and dark modes.

grjones · 2024-04-17T19:04:43Z

I was trying out this exciting branch and ran into this error when running a test:

<...>/lib/python3.12/site-packages/django/db/models/lookups.py:30: in __init__
    self.rhs = self.get_prep_lookup()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = TupleIn(<django.db.models.fields.composite.Cols object at 0x107560980>, <django.db.models.sql.query.Query object at 0x1074e23f0>)

    def get_prep_lookup(self):
        if not isinstance(self.lhs, Cols):
            raise ValueError(
                "The left-hand side of the 'in' lookup must be an instance of Cols"
            )
        if not isinstance(self.rhs, Iterable):
>           raise ValueError(
                "The right-hand side of the 'in' lookup must be an iterable"
            )
E           ValueError: The right-hand side of the 'in' lookup must be an iterable

The issue stems from the use of isnull like so:

MyModel.objects.filter(
    type_override__severity__isnull=False
).update(severity="high")

Curious if anyone ran into this as well.

Edited for traceback:

<...>
lib/python3.12/site-packages/django/db/models/sql/compiler.py:2080: in pre_sql_setup
    self.query.add_filter("pk__in", query)
lib/python3.12/site-packages/django/db/models/sql/query.py:1601: in add_filter
    self.add_q(Q((filter_lhs, filter_rhs)))
lib/python3.12/site-packages/django/db/models/sql/query.py:1617: in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
lib/python3.12/site-packages/django/db/models/sql/query.py:1649: in _add_q
    child_clause, needed_inner = self.build_filter(
lib/python3.12/site-packages/django/db/models/sql/query.py:1563: in build_filter
    condition = self.build_lookup(lookups, col, value)
lib/python3.12/site-packages/django/db/models/sql/query.py:1393: in build_lookup
    lookup = lookup_class(lhs, rhs)
lib/python3.12/site-packages/django/db/models/lookups.py:30: in __init__
    self.rhs = self.get_prep_lookup()

So, this is part of SQLUpdateCompiler and is coming from the update code path.

csirmazbendeguz · 2024-04-18T02:58:00Z

Thanks for testing and reporting the issue @grjones! Indeed, I forgot to handle this use case. I'll look into it this week.

csirmazbendeguz · 2024-04-20T00:52:41Z

@grjones, FYI I pushed the fix

grjones · 2024-04-20T01:25:37Z

@grjones, FYI I pushed the fix

Nice! I hope this gets merged in soon. Your branch has been working great for me.

grjones · 2024-04-22T16:54:36Z

I may have found one other small issue. When adding a regular primary_key=True on a single field, a unique constraint is added. But when using this branch, it becomes an IntegrityError instead. Adding a UniqueConstraint on the composite fields is a work-a-round but ideally would be captured in this PR. Imo, this PR is sooooo close. I'm excited for it to be merged in.

csirmazbendeguz · 2024-04-23T04:44:18Z

@grjones , thanks, I appreciate the feedback, I'll look into it. If a model defines Meta.primary_key, defining primary_key=True on a field should not be possible - could you give me a code example so I know how to reproduce the issue? I didn't know Django added unique constraints to primary keys, I'll check, but isn't that redundant?

grjones · 2024-04-23T05:39:35Z

@grjones , thanks, I appreciate the feedback, I'll look into it. If a model defines Meta.primary_key, defining primary_key=True on a field should not be possible - could you give me a code example so I know how to reproduce the issue? I didn't know Django added unique constraints to primary keys, I'll check, but isn't that redundant?

I'll see if I can give you a solid failing test. My "unique constraint" phrasing might not be exactly right. But ultimately, I believe Django queries the DB first to see if the new object's PK already exists and throws a validation error. The composite key logic doesn't seem to be doing that and so an unhandled IntegrityError is raised instead.

csirmazbendeguz · 2024-05-01T08:48:18Z

@grjones , sorry for the late reply, I've been busy last week. Could you give me more specifics? What's the error message you expect?

grjones · 2024-05-02T03:09:49Z

@grjones , sorry for the late reply, I've been busy last week. Could you give me more specifics? What's the error message you expect?

Actually, I think it's mostly ok. I was using Django Spanner and it's just not quite working with composite keys and will need to be fixed there. I wrote this and it passed. It probably shouldn't say Id though?

from django.core.exceptions import ValidationError
from django.test import TestCase

from .models import Tenant, User


class CompositePKCleanTests(TestCase):
    """
    Test the .clean() method of composite_pk models.
    """

    @classmethod
    def setUpTestData(cls):
        cls.tenant = Tenant.objects.create()

    def test_validation_error_is_raised_when_pk_already_exists(self):
        test_cases = [
            {"tenant": self.tenant, "id": 2412, "email": "user2412@example.com"},
            {"tenant_id": self.tenant.id, "id": 5316, "email": "user5316@example.com"},
            {"pk": (self.tenant.id, 7424), "email": "user7424@example.com"},
        ]
        expected = "{'id': ['User with this Id already exists.']}"
        for fields in test_cases:
            User.objects.create(**fields)
            with self.assertRaisesMessage(ValidationError, expected):
                User(**fields).clean()

LilyFoote

This is a great start!

I've left a bunch of ideas for improvement. Feel free to push back if you think I'm wrong about anything.

django/db/models/base.py

django/db/models/fields/composite.py

tests/composite_pk/test_get.py

tests/composite_pk/test_update.py

tests/composite_pk/tests.py

csirmazbendeguz · 2024-05-04T03:31:50Z

Thank you so much for taking the time to review my changes @LilyFoote !
I have two questions:

If Meta.primary_key is defined, this PR will automatically add a composite field called primary_key to the model. What do you think about this approach? I felt like it was easier to handle the composite primary keys this way as we can run checks against the meta class instead of traversing the model's fields for a composite field.
I wrote a lot of tests testing the underlying queries made by the ORM. It makes a lot of sense to me, but I haven't seen this type of tests that much in the Django source code - do these tests look okay to you?

LilyFoote · 2024-05-04T12:13:30Z

If Meta.primary_key is defined, this PR will automatically add a composite field called primary_key to the model. What do you think about this approach?

I don't feel strongly that this is better or worse than another option here, so happy to go with what you think is best.

I wrote a lot of tests testing the underlying queries made by the ORM. It makes a lot of sense to me, but I haven't seen this type of tests that much in the Django source code - do these tests look okay to you?

I like your tests quite a bit - they're pretty readable and comprehensive. The main issue I have with them is that they're written for specific databases instead of for generic database features. Where possible Django strongly prefers to test based on features because then the tests apply to as many databases as possible (including third party database libraries). I think the asserts of the actual SQL might be a bit tricky to adapt though, so we might need a different way to check what they're checking.

Also, after I reviewed yesterday, I thought of some more things:

We should add migrations tests to make sure that adding/removing Meta.primary_key works correctly and that removing a field that's part of a primary key also does something appropriate.
We might want tests for composite keys in forms and the admin. Maybe there's other areas too that we need to check the interactions.

charettes · 2024-05-05T19:51:19Z

tests/composite_pk/models/tenant.py

+    user = models.ForeignObject(
+        User,
+        on_delete=models.CASCADE,
+        from_fields=("tenant_id", "user_id"),
+        to_fields=("tenant_id", "id"),
+        related_name="+",
+    )


yeah I think this should be a stretch goal to get it working. See the comment above about MultiColSource.

charettes · 2024-05-05T19:51:35Z

django/db/models/fields/composite.py

+        return compiler.compile(WhereNode(exprs, connector=OR))
+
+
+class Cols(Expression):


I wonder if there is an opportunity to merge this TuplesIn, Cols, and friends logic with MultiColSource so it's less of an 👽. They both do very similar thing.

Thanks @charettes , I'll need to look into this, I wasn't aware.

I merged Cols with MultiColSource (ad51da4) however, I'm not sure this is correct.

As far as I understand, MultiColSource was meant to represent columns in a JOIN, and as such, it has a sources field. Cols, on the other hand, was meant to represent a list of columns and it doesn't need a sources field. WDYT?

I reverted back to Cols. Please resolve if you agree.

…ion errors

apollo13 · 2024-06-08T08:54:30Z

django/db/backends/base/schema.py

+        # to the table definition.
+        # It's expected 'primary_key=True' isn't set on any fields (see E043).
+        pk = model._meta.pk
+        if hasattr(pk, "columns"):


This might have been answered somewhere else already, so please ignore the question if it was. I am wondering if it would make sense to handle single column primary keys like this as well. What would be the upside of having different objects (?) here for single vs multi-columns fields? Or do we gain some backwards compat wins here by having it like this?

I don't think we've considered this yet. My feeling without digging in is that it's not necessary for this feature, but it might be a nice follow-up refactor.

Ok, if we want to do it in a follow-up refactor we should still ensure that it ends up in the same release. While Meta is probably not documented, changes there will still hurt more advanced 3rd party packages.

The only difference is, primary_key=True creates the primary key inline, while Meta.primary_key creates it as a table level constraint.

CREATE TABLE foo ( id INTEGER PRIMARY KEY )

vs.

CREATE TABLE foo ( id INTEGER, PRIMARY KEY (id) )

Afaik all db backends support both inline and table-level primary keys, so there's no real difference in practice. It would take some work to move away from inline primary keys and it's out of scope for this ticket.

Actually, I am second guessing if Meta.primary_key is really necessary now.

You do raise the entirely valid point, why is it None if primary_key=True is set on a field? As a user, I would expect it to always store some info related to the primary key. And I would expect this info to be the source of truth.

So, how about we consider the following API instead?

class Foo(models.Model): primary_key = models.CompositePrimaryKey("tenant_id", "id") tenant_id = models.IntegerField() id = models.IntegerField()

This is how @LilyFoote did it too in the previous PR, I just thought adding a Meta option is more user friendly.
But if it leads to confusion, I'm not sure it's worth it anymore.

Yes, that explicit approach certainly feels less magical (ie we don't have to add an extra field to the class and the user can name it whatever they want). All in all it feels more consistent.

Yeah, this feels like the right API to me given this discussion.

I adjusted it 👍

Let me know what you think, and thanks for the review @apollo13 , I appreciate it! ❤️

django/db/models/options.py

omerfarukabaci · 2024-06-10T16:27:27Z

Hey @csirmazbendeguz, thank you for the amazing work out there! I was trying to test this branch on my local with SQLite and realised a few things:

If you run makemigrations for a model with a CompositePrimaryKey, the resulting migration file has erroneous imports. To fix this, I believe we need to add django.db.models.fields.composite path to the if...elif block here.

Assume that I have the following models:

class Author(models.Model):
name = models.CharField(max_length=100)

class Book(models.Model):
    id = models.CompositePrimaryKey("author", "title")
    author = models.ForeignKey(Author, on_delete=models.CASCADE, related_name="books")
    title = models.CharField(max_length=255)

With the current implementation, following test fails:

class TestCompositeFks(TestCase):
    def test_composite_fks(self):
        author = Author.objects.create(name="Author")
        book = Book.objects.create(author=author, title="Title")
        list(Author.objects.filter(books__in=[book])) == book

with an OperationalError, caused by a syntax error. Executed SQL is as following:

SELECT
    "books_author"."id",
    "books_author"."name"
FROM
    "books_author"
    INNER JOIN "books_book" ON ("books_author"."id" = "books_book"."author_id")
WHERE
    "books_book"."author_id", "books_book"."title" IN ((1, 'Title'))

because LHS in WHERE clause should have been wrapped with parantheses like this:

...
WHERE
    ("books_book"."author_id", "books_book"."title") IN ((1, 'Title'))

Unfortunately I didn't have a time to deep-dive to this.

Not a big issue but my code editor (VSCode) does not recognize models.CompositePrimaryKey, although the import is working fine. This is probably related with Pylance or something that VSCode uses to recognize fields under models module.

Again thanks for this amazing initiative! 🚀

omerfarukabaci · 2024-06-10T16:34:06Z

django/db/models/fields/composite.py

+    def get_lookup(self, lookup_name):
+        if lookup_name == "exact":
+            return TupleExact
+        elif lookup_name == "gt":
+            return TupleGreaterThan
+        elif lookup_name == "gte":
+            return TupleGreaterThanOrEqual
+        elif lookup_name == "lt":
+            return TupleLessThan
+        elif lookup_name == "lte":
+            return TupleLessThanOrEqual
+        elif lookup_name == "in":
+            return TupleIn
+
+        raise NotImplementedError


Is there a specific reason to not to use CompositePrimaryKey.register_lookup as we do for other fields (e.g. JSONField)?

csirmazbendeguz · 2024-06-10T17:42:49Z

Thanks a lot for the review @omerfarukabaci ! I'll take a look

csirmazbendeguz · 2024-06-11T06:04:55Z

Author.objects.filter(books__in=[book])

@omerfarukabaci , I pushed the changes to support this, but note that filtering on reverse relations is one of those "gotchas" in Django, it may not produce the results you expect.

EDIT: I mean it might return duplicates, you probably already know this, I'm just mentioning it just in case.

csirmazbendeguz · 2024-06-11T09:27:49Z

If you run makemigrations for a model with a CompositePrimaryKey, the resulting migration file has erroneous imports

Yes, I recently changed the API to CompositePrimaryKey, the migrations are not 100% yet. I'm working on sorting them out.
I pushed the fix for the issue you mentioned, thanks 👍

omerfarukabaci · 2024-06-11T14:00:13Z

@csirmazbendeguz Thanks for your answers, now the above issues seem like fixed, created migration is correct and reverse relation lookup is working as expected. Thank you! 🚀

While I was testing it further with the exact same models, I realized another issue:

class TestCompositeFks(TestCase):
    def test_composite_fks(self):
        author = Author.objects.create(name="Author")
        Book.objects.create(author=author, title="Title")
        author = Author.objects.annotate(book_count=Count("books")).get()
        assert author.book_count == 1

This test fails with the following error:

django.db.utils.OperationalError: wrong number of arguments to function COUNT()

Executed SQL is as following:

SELECT
    "books_author"."id",
    "books_author"."name",
    COUNT("books_book"."author_id", "books_book"."title") AS "book_count"
FROM
    "books_author"
    LEFT OUTER JOIN "books_book" ON ("books_author"."id" = "books_book"."author_id")
GROUP BY
    "books_author"."id",
    "books_author"."name"

If we could change the parameter we pass to the COUNT function to a concatenation as below:

COUNT("books_book"."author_id" || '-' || "books_book"."title")

it should work fine (if I am not missing something), with the exception that for some databases we need to use CONCAT function instead of || operator, which might be resolved using the existing db.models.functions.Concat function.

Note: I am not sure if concatenation works between every data type that is allowed to be a primary key, although this could be considered as an edge case.

csirmazbendeguz · 2024-06-11T16:01:16Z

Thanks @omerfarukabaci , these bug reports are very helpful. Yes, I haven't considered annotations with multi-column pks. I'll look into this.

csirmazbendeguz mentioned this pull request Apr 7, 2024

Fixed #373 -- Added CompositeField-based Meta.primary_key. #18031

Closed

csirmazbendeguz force-pushed the ticket_373 branch 2 times, most recently from 6a26b19 to c75dcdd Compare April 19, 2024 12:22

Fixed django#373 -- Added CompositeField-based Meta.primary_key.

4be1c68

csirmazbendeguz force-pushed the ticket_373 branch from c75dcdd to 4be1c68 Compare April 20, 2024 00:12

LilyFoote reviewed May 3, 2024

View reviewed changes

charettes reviewed May 5, 2024

View reviewed changes

csirmazbendeguz added 12 commits May 8, 2024 18:53

Fixed django#373 -- Add field to error message, simplify tuple operat…

cee213d

…ion errors

Fixed django#373 -- Rename CompositeField to CompositePrimaryKey

e2d8868

Fixed django#373 -- Add __len__ to CompositePrimaryKey

259605a

Fixed django#373 -- Add some docs

9e01443

Fixed django#373 -- Fix docs formatting

fa4af00

Fixed django#373 -- Add more docs

8354bd0

Fixed django#373 -- Fix docs

d20f8e0

Fixed django#373 -- Fix integrity error tests

dcd78fc

Fixed django#373 -- Remove unnecessary asserts

3576dec

Fixed django#373 -- Make a test simpler

1ef76e4

Fixed django#373 -- Make get tests better

3d0dcf3

Fixed django#373 -- Fix test

6a2bb92

Fixed django#373 - Use fk name in test model

fa5fe70

csirmazbendeguz mentioned this pull request Jun 6, 2024

[WIP] Fixed #373 - Added composite foreign keys. #18205

Draft

6 tasks

csirmazbendeguz added 2 commits June 6, 2024 18:11

Fixed django#373 - Empty commit

ca347a9

Fixed django#373 - Empty commit

ae3c737

apollo13 reviewed Jun 8, 2024

View reviewed changes

django/db/models/options.py Outdated Show resolved Hide resolved

csirmazbendeguz added 7 commits June 9, 2024 13:31

Fixed django#373 - Rename primary_key field to _primary_key

fef2ee4

Fixed django#373 - Small adjustment

bdff86c

Fixed django#373 - Randomize pk field name

1d900d1

Fixed django#373 - Add comment

29fd7c1

Fixed django#373 - Better name

ac463a0

Fixed django#373 - Revert random pk name

ddc07ff

Fixed django#373 - Remove Meta.primary_key option

3ca41fe

csirmazbendeguz changed the title ~~Fixed #373 -- Added CompositePrimaryKey-based Meta.primary_key.~~ Fixed #373 -- Added CompositePrimaryKey. Jun 10, 2024

csirmazbendeguz added 3 commits June 10, 2024 21:05

Fixed django#373 - Refactor unnest_composite_fields

2626948

Fixed django#373 - Adjust docs

ee09f8b

Fixed django#373 - Adjust docs

1b88f15

omerfarukabaci suggested changes Jun 10, 2024

View reviewed changes

csirmazbendeguz added 4 commits June 11, 2024 12:11

Fixed django#373 - Add support for reverse lookups

98ba3af

Fixed django#373 - Add support for lt, lte, gt, gte on reverse lookups

5d85922

Fixed django#373 - Add support for tuple isnull lookups

3ca3502

Fixed django#373 - Remove unused var

40dc827

csirmazbendeguz mentioned this pull request Jun 11, 2024

Fixed #27452 -- Added serial fields to contrib.postgres. #18123

Open

6 tasks

Fixed django#373 - Fix composite pk field import

0fe7750

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed #373 -- Added CompositePrimaryKey. #18056

Fixed #373 -- Added CompositePrimaryKey. #18056

csirmazbendeguz commented Apr 7, 2024 •

edited

grjones commented Apr 17, 2024 •

edited

csirmazbendeguz commented Apr 18, 2024 •

edited

csirmazbendeguz commented Apr 20, 2024

grjones commented Apr 20, 2024

grjones commented Apr 22, 2024

csirmazbendeguz commented Apr 23, 2024

grjones commented Apr 23, 2024

csirmazbendeguz commented May 1, 2024 •

edited

grjones commented May 2, 2024 •

edited

LilyFoote left a comment

csirmazbendeguz commented May 4, 2024

LilyFoote commented May 4, 2024

charettes May 5, 2024

charettes May 5, 2024

csirmazbendeguz May 8, 2024

csirmazbendeguz May 11, 2024 •

edited

csirmazbendeguz May 23, 2024

apollo13 Jun 8, 2024

LilyFoote Jun 8, 2024

apollo13 Jun 8, 2024

csirmazbendeguz Jun 8, 2024 •

edited

csirmazbendeguz Jun 8, 2024

csirmazbendeguz Jun 10, 2024 •

edited

apollo13 Jun 10, 2024

LilyFoote Jun 10, 2024

csirmazbendeguz Jun 10, 2024

csirmazbendeguz Jun 10, 2024

omerfarukabaci commented Jun 10, 2024 •

edited

omerfarukabaci Jun 10, 2024

csirmazbendeguz commented Jun 10, 2024

csirmazbendeguz commented Jun 11, 2024 •

edited

csirmazbendeguz commented Jun 11, 2024

omerfarukabaci commented Jun 11, 2024

csirmazbendeguz commented Jun 11, 2024

		return compiler.compile(WhereNode(exprs, connector=OR))


		class Cols(Expression):

Fixed #373 -- Added CompositePrimaryKey. #18056

Are you sure you want to change the base?

Fixed #373 -- Added CompositePrimaryKey. #18056

Conversation

csirmazbendeguz commented Apr 7, 2024 • edited

Trac ticket number

Branch description

Checklist

grjones commented Apr 17, 2024 • edited

csirmazbendeguz commented Apr 18, 2024 • edited

csirmazbendeguz commented Apr 20, 2024

grjones commented Apr 20, 2024

grjones commented Apr 22, 2024

csirmazbendeguz commented Apr 23, 2024

grjones commented Apr 23, 2024

csirmazbendeguz commented May 1, 2024 • edited

grjones commented May 2, 2024 • edited

LilyFoote left a comment

Choose a reason for hiding this comment

csirmazbendeguz commented May 4, 2024

LilyFoote commented May 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csirmazbendeguz May 11, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csirmazbendeguz Jun 8, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csirmazbendeguz Jun 10, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omerfarukabaci commented Jun 10, 2024 • edited

Choose a reason for hiding this comment

csirmazbendeguz commented Jun 10, 2024

csirmazbendeguz commented Jun 11, 2024 • edited

csirmazbendeguz commented Jun 11, 2024

omerfarukabaci commented Jun 11, 2024

csirmazbendeguz commented Jun 11, 2024

csirmazbendeguz commented Apr 7, 2024 •

edited

grjones commented Apr 17, 2024 •

edited

csirmazbendeguz commented Apr 18, 2024 •

edited

csirmazbendeguz commented May 1, 2024 •

edited

grjones commented May 2, 2024 •

edited

csirmazbendeguz May 11, 2024 •

edited

csirmazbendeguz Jun 8, 2024 •

edited

csirmazbendeguz Jun 10, 2024 •

edited

omerfarukabaci commented Jun 10, 2024 •

edited

csirmazbendeguz commented Jun 11, 2024 •

edited