Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support TorchArrow column with binary type (SQL VARBINARY, pyarrow.binary) #178

Open
scotts opened this issue Feb 4, 2022 · 0 comments
Open

Support TorchArrow column with binary type (SQL VARBINARY, pyarrow.binary) #178

scotts opened this issue Feb 4, 2022 · 0 comments

Comments

@scotts
Copy link
Contributor

scotts commented Feb 4, 2022

cc: @wenleix
YLGH pushed a commit to YLGH/torcharrow that referenced this issue May 7, 2022
@joshuadeng @facebook-github-bot
Convert input/output in stats to be MB per iteration (pytorch#178)
b121fab 
Summary:
Pull Request resolved: pytorch/torchrec#178

Instead of naively showing input as just the pooling factor and output as just the embedding dimension
this diff changes planner stats to use the actual size of input & output in terms of megabytes per iteration

**input**: global_batch_size * pooling factor * sizeof(dtype of input)

**output**: global_batch_size * (output size (1 in pooled)) * sizeof(dytpe of emb) * emb_dim

This provides a sense of scale for data coming in and out, and additionally makes plans with multiple sharding types directly comparable.

Also fixes a bug with TWCW, we incorrectly specified the ranks as entire world size when it should be limited to the local world of the  host that the parameter is sharded on.

Reviewed By: dstaay-fb

Differential Revision: D35153224

fbshipit-source-id: c1e7d717ec0c1d074f7e059d843fba2d287eee56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@scotts