Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update endpoint for dag size result on app/user updates #603

Open
walkah opened this issue Jun 7, 2022 · 1 comment
Open

Update endpoint for dag size result on app/user updates #603

walkah opened this issue Jun 7, 2022 · 1 comment
Labels
🐛 bug Something isn't working

Comments

@walkah
Copy link
Member

walkah commented Jun 7, 2022

Summary

Problem

The fission-server is currently using a deprecated IPFS API to size usage apps and user file systems.

Impact

This means there is incorrect reporting in the database for how much space an app or user is using.

Solution

We should replace the call to the (deprectaed) /api/v0/object/stat endpoint (https://docs.ipfs.io/reference/http/api/#api-v0-object-stat) with a version that is "DAG" aware (that isn't re-counting common blocks in the DAG). The documentation recommends /api/v0/files/stat (but /api/v0/dag/stat might be more appropriate?).

See also: fission-codes/kubo#56

@walkah walkah added the 🐛 bug Something isn't working label Jun 7, 2022
@justindotpub
Copy link

Looks like /api/v0/dag/stat is the one we want, though it has performance implications since it traverses the DAG, skipping duplicates.

# one file in a directory, therefore no dupe files
»  random-files --dirs 0 --files 1 --filesize 4096 blah
blah/rnlq6zl7wr11

»  ipfs add -r blah
added QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2 blah/rnlq6zl7wr11
added Qma5pU4piKaN84xzafFK8QGEZYThcDSqpQLptGLod5qVFK blah
 3.49 KiB / 3.49 KiB [==========================================================================================================] 100.00%

# file
»  ipfs dag stat QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2                              
Size: 3585, NumBlocks: 1

»  ipfs files stat /ipfs/QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2
QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2
Size: 3574
CumulativeSize: 3585
ChildBlocks: 0
Type: file

# directory
»  ipfs dag stat Qma5pU4piKaN84xzafFK8QGEZYThcDSqpQLptGLod5qVFK                                              
Size: 3644, NumBlocks: 2

»  ipfs files stat /ipfs/Qma5pU4piKaN84xzafFK8QGEZYThcDSqpQLptGLod5qVFK
Qma5pU4piKaN84xzafFK8QGEZYThcDSqpQLptGLod5qVFK
Size: 0
CumulativeSize: 3644
ChildBlocks: 1
Type: directory

# two file in a directory, both files dupes with diff names.
»  cp blah/rnlq6zl7wr11 blah/rnlq6zl7wr11-dupe1        

»  ipfs add -r blah                                                    
added QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2 blah/rnlq6zl7wr11
added QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2 blah/rnlq6zl7wr11-dupe1
added QmTPQPCBWHS9SSo2QKErK87LzJcJMw2PaffWfP9dKTBPer blah
 6.98 KiB / 6.98 KiB [=================================================================================================================================================================================================================================================================] 100.00%

# file one and two have same CID, so same output here
»  ipfs dag stat QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2                                              
Size: 3585, NumBlocks: 1

»  ipfs files stat /ipfs/QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2                                              
QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2
Size: 3574
CumulativeSize: 3585
ChildBlocks: 0
Type: file

# dag stat of directory is only slightly larger than last time, still with 2 blocks
»  ipfs dag stat QmTPQPCBWHS9SSo2QKErK87LzJcJMw2PaffWfP9dKTBPer                                              
Size: 3705, NumBlocks: 2

# files stat of dir is twice size of file (7,148), plus 142 bytes for metadata?
»  ipfs files stat /ipfs/QmTPQPCBWHS9SSo2QKErK87LzJcJMw2PaffWfP9dKTBPer                                              
QmTPQPCBWHS9SSo2QKErK87LzJcJMw2PaffWfP9dKTBPer
Size: 0
CumulativeSize: 7290
ChildBlocks: 2
Type: directory

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐛 bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants