Resolve Parquet shard count via bucket index to optimize storage calls#7648
Resolve Parquet shard count via bucket index to optimize storage calls#7648SungJin1212 wants to merge 2 commits into
Conversation
SungJin1212
force-pushed
the
parquet-shard-count-from-bucket-index
branch
from
f7b225e to
0fdab66
Compare
| shardCounts := make(map[string]int, len(blockIDs)) | ||
|
|
||
| if p.bucketIndexEnabled { | ||
| idx, err := bucketindex.ReadIndex(ctx, p.indexBucket, p.userID, p.limits, p.logger) |
There was a problem hiding this comment.
This should be cached with some TTL instead? Or we would rather have a separate goroutine to sync bucket index periodically rather than resolving it at query time.
There was a problem hiding this comment.
If we use bucketindex.Loader it has built-in caching
SungJin1212
force-pushed
the
parquet-shard-count-from-bucket-index
branch
from
0fdab66 to
641b69d
Compare
…e calls Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
SungJin1212
force-pushed
the
parquet-shard-count-from-bucket-index
branch
from
6656f19 to
2f7ecc9
Compare
|
@yeya24
|
|
I wonder how this bucket index integration works with the Store gateway hybrid mode mentioned in #7140. Do we imagine the hybrid mode to be as part of TSDB bucket store, or Parquet bucket store or a new federated store. |
|
@yeya24 |
2 participants
What this PR does:
This PR updates the Parquet shard resolution logic to utilize the bucket index, reducing the number of object storage calls.
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]docs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flags