Introduce custom hash table data structures.

[chandlerc]

The hash table design is heavily based on Abseil's "Swiss Tables" design. It uses an array of bytes storing metadata about each entry and an array of entries where each is a pair of key and value. The metadata byte consists of 7-bits of hash of the key (distinct from the bits used to index the table), and one bit indicating the presence of a special entry -- either empty or deleted.

There are a large range of optimizations and other nuanced aspects of this hash table design and implementation, a good point to understand that context is raw_hashtable.h which has an overview of the design and references to various other files for relevant details.

[josh11b]

I wanted to give feedback on map.h early, before you update set.h

[chandlerc]

Thanks, definitely useful to get map.h tidied up before I port it to set.h. I think I've responded to all the comments there.

[josh11b]

Sending what comments I have so far.

[chandlerc]

Thanks for all the feedback so far, PTAL.

[josh11b]

Finished a pass through raw_hashtable.h

[chandlerc]

PTAL!

Beyond inline replies, also was able to simplify the specialization strategy for the table type and improved some other comments spotted as I was working.

Also, think this is far enough along in review I'm moving it out of draft.

[chandlerc]

PTAL!

Beyond inline replies, also was able to simplify the specialization strategy for the table type and improved some other comments spotted as I was working.

Also, think this is far enough along in review I'm moving it out of draft.

[chandlerc]

So, I'm most of the way through adding really nice support for handling stateful keys like indices into vectors and such. It turned out to be necessary to even move the toolchain over to these data structures as it also lets us use something other than operator== for equality, and we need that for APFloat keys. There are a few interesting things that emerged:

• I've ended up removing unnecessary functionality that also makes the stateful case unimplementable.

• I've found and cleaned up a number of innocuous inconsistencies in the code.

• I've been able to implement the test you asked for where we control the hashing and force every key to collide. I can implement even more of these if desired.

Wanted to both let you know @josh11b about my progress there, and also ask whether you'd like me to merge this into this code review or keep it as a follow-up. I can easily manage either way.

[josh11b]

So, I'm most of the way through adding really nice support for handling stateful keys like indices into vectors and such. It turned out to be necessary to even move the toolchain over to these data structures as it also lets us use something other than operator== for equality, and we need that for APFloat keys. There are a few interesting things that emerged:

• I've ended up removing unnecessary functionality that also makes the stateful case unimplementable.
• I've found and cleaned up a number of innocuous inconsistencies in the code.
• I've been able to implement the test you asked for where we control the hashing and force every key to collide. I can implement even more of these if desired.

Wanted to both let you know @josh11b about my progress there, and also ask whether you'd like me to merge this into this code review or keep it as a follow-up. I can easily manage either way.

I'm fine either way. If it simplifies the files haven't reviewed yet, it would be a benefit.

[chandlerc]

So, I'm most of the way through adding really nice support for handling stateful keys like indices into vectors and such. It turned out to be necessary to even move the toolchain over to these data structures as it also lets us use something other than operator== for equality, and we need that for APFloat keys. There are a few interesting things that emerged:

• I've ended up removing unnecessary functionality that also makes the stateful case unimplementable.
• I've found and cleaned up a number of innocuous inconsistencies in the code.
• I've been able to implement the test you asked for where we control the hashing and force every key to collide. I can implement even more of these if desired.

Wanted to both let you know @josh11b about my progress there, and also ask whether you'd like me to merge this into this code review or keep it as a follow-up. I can easily manage either way.

I'm fine either way. If it simplifies the files haven't reviewed yet, it would be a benefit.

After discussion, it seems a bit borderline with pros and cons, but on the whole probably simplifies the review. Added and pushed. I did make it a clean follow-up commit so that it is easy to see the diff: 807de9e (#3940)

[josh11b]

Finished a pass on raw_hashtable_metadata_group.h

[josh11b]

I suspect many of the comments about set.h also apply to map.h

[josh11b]

Read through set_test.cpp and started on map_test.cpp

[josh11b]

I've now looked at everything except for files with names containing "benchmark"

[josh11b]

Finished a complete pass

[chandlerc]

Addressed most of the metadata_group stuff. Sorry my search-and-replace for mask with bits missed so much, I think I must have messed up that search really badly. =[

Still need to go through comments on other benchmark files, but figured I'd post these updates.

[chandlerc]

Ok, I think I got all the benchmark comments now as well as a collection of other comments I had missed that were still open. PTAL!

[josh11b]

There is still a "TODO" in the PR summary, though hopefully that has been largely addressed by now.

[chandlerc]

Huge thanks to Josh for all the review on this ... really extensive PR. Also thanks to all the folks who gave feedback on the design or approach or discussed!