Go API code.

[bruno-ga]

Hello. I am working on writing Go wrapper code around the generated flatbuffer types. I am mainly looking at the C++ and Java implementations (they seem to be pretty close to each other) but it seems both only support a static NodeItem that can not have any payload data associated with it. Is there really the case? Is there any implementation that does support extra data to be stored and retrieved? I will definitely need that in the Go wrapper and it seems I would be willing to roll out my own custom version of that but in case there is already a specification for this, I can follow it.

Any help here would be appreciated.

[bjornharrtell]

I don't think it's a good idea to wrap Go around C++ or Java. It would be better to use the generated Go source directly, which lives at https://github.com/flatgeobuf/flatgeobuf/tree/master/src/go and reimplement the higher level APIs as needed. It's not that much C++ code to port and would avoid wrapper issues.

[bruno-ga]

Sorry, I guess I was not clear. I am writing the Go code around the generated Go flatbuffers (i.e. I am "wrapping" the generated flatbuffers just like the other languages do). I am just using the Java and C++ code as a template. I am not wrapping Java or C++ code. So I am doing exactly what you suggested.

My question still stands though. It seems that neither the C++ or Java code include support for fetching data associated with a NodeItem. Ideally, in Go I would like to have a generic NodeItem that had a generic data member. Than can be easily done as long as I can figure out how to read and write that data. This is the information I am looking for.

Thanks for your reply in any case.

[bruno-ga]

Changed the title of the discussion to avoid potential misinterpretation.

[bjornharrtell]

I see, then it's welcome contribution!

However it sounds to be that you might be misunderstanding NodeItem. Assuming you mean NodeItem in the spatial index (packed R-Tree) it cannot be generic. The spatial index of FlatGeobuf is entirely static. A NodeItem contains an offset to a Feature in the data section which can be seen as the generic "data" record associated with the tree node item. The static design of the spatial index was by design to make it simple and fast (influenced by flatbush) - I can understand if that seems restrictive but it was what I decided.

[bruno-ga]

You are correct. It was a wrong interpretation I made out of trying to figure it out from the source code. Now I am not only fully aware of how it works, I have a fully working Go implementation with a higher level API that not only allows writing and using fgb files, it also make it easy to create converters from potentially any relevant format (I have one for shapefiles) to flatgeobuf in what I hope to be a very simple way.

I intend to contribute it back soon but I wrote it as part of a project for my company so I need to go through the proper channels before being able to contribute the code back.

[michaelkirk]

I'm not sure if this is what you were thinking, but in the past I've wondered if storing "aggregate" statistics on index nodes would be useful.

My specific use case was displaying on a map of a lot of individual incidents, which, when zoomed in would show individual points, but when zoomed out would summarize the data as a heatmap.

The file was very large (lots of points). When zoomed in, it worked well to utilize FGB's bbox filtering to get just the points on the viewport, but when zoomed out, I had no good way to build the heatmap summarizing the points. Downloading the entire FGB was not feasible because it was very large.

At the time I was thinking if I could store some kind of aggregate statistics on the index nodes, I could use that for rendering the heat map portion.

There is already a configurable "node size" as a parameter in the header, but it's interpreted to mean the "branching factor", so just trying to use that to add "padding" for metadata in the index nodes wouldn't work.

You could add a new header parameter like "index node payload size (default = 0)" for a fixed metadata chunk in each node, or add some kind of pointer to externally stored metadata, but it would complicate the format quite a bit, and I'm not sure how many use cases there actually are for this kind of bbox aggregations.

[bruno-ga]

@michaelkirk that is a more contrived form of what I wanted to do but, in general, yes, that would be it. But I guess I was just missing a part of the puzzle.

[bruno-ga]

Ok, I guess there was a misunderstanding in my part. So the idea is that the index is literally just that, but it does include the offset that can be set per NodeItem and then can be used to refer to whatever data (in a separate file?).

So the idea would be generate that file and collect the offsets and then use that to generate the index, right?

[bruno-ga]

Assuming my last comment reflects (more or less, maybe) how things would be, I guess one can actually create an extended API that would encapsulate both the index and data and allow for easy retrieval of data based on the hits in the RTree.

[brunoga]

After playing a bit more with the code and specification I now have the full PackedRTree implementation working in go and have most of the fbg generation code done.

The next big thing is actually being able to use the fbs file by mmaping it then doing queries in the RTree and then reasdinmg the associated data in the fbg file. Is there some sample code somewhere that shows yje canonical way of doing it? I guess I found some javascript example but I would be more in my turf if I could find a C++ or java version of it.

Thanks in advance.

[bruno-ga]

FWIIW, I just submitted a pull request for the Go implementation.

#303