Post
89
The hf article detailing the integration between hf datasets and dagster is live 🤗 : https://huggingface.co/blog/AINovice2005/dagster-hf-datasets
TheHFStack
community
AI & ML interests
Infrastructure, integrations and tooling for the Hugging Face ecosystem.
Recent Activity
AINovice2005
posted an update 6 days ago
AINovice2005
published a
bucket 7 days ago
AINovice2005
updated a
Space 7 days ago
AINovice2005
posted an update 9 days ago
Post
69
Excited to share the release of dagster-hf-datasets: A Dagster-native integration that brings Hugging Face Datasets into Dagster's asset-oriented orchestration model
The integration enables:
• 🤗 Dataset and DatasetDict assets
• 🐙 Dagster asset lineage and observability
• 📦 Parquet-backed materialization via HFParquetIOManager
• 🚀 Publishing curated datasets back to the Hugging Face Hub
• 📝 Automatic dataset card generation from pipeline metadata
As the Hub continues to grow beyond 1M+ datasets, orchestration, reproducibility, and observability are becoming increasingly important parts of the dataset lifecycle. I'm also working on a longer article covering the architecture and data pipelines enabled by the integration.
More Soon!
https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets
https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets/docs
The integration enables:
• 🤗 Dataset and DatasetDict assets
• 🐙 Dagster asset lineage and observability
• 📦 Parquet-backed materialization via HFParquetIOManager
• 🚀 Publishing curated datasets back to the Hugging Face Hub
• 📝 Automatic dataset card generation from pipeline metadata
As the Hub continues to grow beyond 1M+ datasets, orchestration, reproducibility, and observability are becoming increasingly important parts of the dataset lifecycle. I'm also working on a longer article covering the architecture and data pipelines enabled by the integration.
More Soon!
https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets
https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets/docs
AINovice2005
published a
Space 23 days ago
AINovice2005
posted an update about 1 month ago
Post
107
Published my first HF Article 🤗 , which discusses about how to benchmark pixel art with LLMs: https://huggingface.co/blog/AINovice2005/pixel-art-bench
AINovice2005
posted an update about 2 months ago
Post
172
I've built a system to make open-source contributions easier to understand across repositories.
It:
aggregates merged external PRs (reviewed by maintainers)
structures them into a single contributions.md
adds a lightweight AI layer to query patterns and impact
The idea is to move from scattered PRs to a readable changelog of work.
Read about it: https://medium.com/@paragekbote23/from-commits-to-impact-building-an-automated-changelog-for-open-source-contributions-20cdfebcee58
It:
aggregates merged external PRs (reviewed by maintainers)
structures them into a single contributions.md
adds a lightweight AI layer to query patterns and impact
The idea is to move from scattered PRs to a readable changelog of work.
Read about it: https://medium.com/@paragekbote23/from-commits-to-impact-building-an-automated-changelog-for-open-source-contributions-20cdfebcee58
AINovice2005
posted an update 3 months ago
Post
3512
In celebration of the new storage graph feature on the Hub, here's mine 😊 :
Post inspired by @ZennyKenny
Post inspired by @ZennyKenny
AINovice2005
posted an update 3 months ago
Post
155
I recently created my first storage bucket to store experiment data of my performance analysis of 15 tokenizers across 20 languages.
The setup is simple enough for a new product and can be scalable depending on the use-case 🤗 .
Bucket: https://huggingface.co/buckets/AINovice2005/tokenizer-benchmark
github gist: https://gist.github.com/ParagEkbote/b3877f667f84cbb9a27bdaca94ba662a
Article: https://medium.com/@paragekbote23/one-sentence-fifteen-tokenizers-a-tokenizer-benchmarking-pipeline-with-hf-storage-buckets-2e59790276fd
The setup is simple enough for a new product and can be scalable depending on the use-case 🤗 .
Bucket: https://huggingface.co/buckets/AINovice2005/tokenizer-benchmark
github gist: https://gist.github.com/ParagEkbote/b3877f667f84cbb9a27bdaca94ba662a
Article: https://medium.com/@paragekbote23/one-sentence-fifteen-tokenizers-a-tokenizer-benchmarking-pipeline-with-hf-storage-buckets-2e59790276fd
AINovice2005
posted an update 3 months ago
Post
153
I'm pleased to announce that native trackio support has landed in optuna v.4.8.0.
Now you can store HPO trial data to the 🤗 Hub as a dataset: AINovice2005/toy-optuna-hpo-dataset
or Space: AINovice2005/optuna-dashboard
Article which talks about the integration: https://medium.com/@paragekbote23/designing-and-contributing-an-optuna-integration-that-performs-well-in-practice-21a07d77ec2c
Now you can store HPO trial data to the 🤗 Hub as a dataset: AINovice2005/toy-optuna-hpo-dataset
or Space: AINovice2005/optuna-dashboard
Article which talks about the integration: https://medium.com/@paragekbote23/designing-and-contributing-an-optuna-integration-that-performs-well-in-practice-21a07d77ec2c
AINovice2005
posted an update 3 months ago
Post
96
Pro tip2: You can treat HF datasets as versioned repos by pinning a specific revision (tag, branch or commit) when downloading files. 🧠
This ensures your data processing pipelines always use the exact dataset state before passing the data to the model. It enables reproducible pipelines and allows for reliable outputs of your ML system.
This ensures your data processing pipelines always use the exact dataset state before passing the data to the model. It enables reproducible pipelines and allows for reliable outputs of your ML system.
from huggingface_hub import hf_hub_download
data_path = hf_hub_download(
repo_id="lysandre/arxiv-nlp",
filename="train.parquet",
repo_type="dataset",
revision="877b84a8f93f2d619faa2a6e514a32beef88ab0a"
)AINovice2005
posted an update 3 months ago
Post
388
Pro tip: If you are finetuning any model with tensorboard logs enabled, be sure to upload them to HF Hub as event artifacts, they can be viewed instantly. 🚀
I previously remembered this done in the notus model release: argilla/notus-7b-v1
Examples:
AINovice2005/ModernBERT-base-lora-cicflow-1m-r8
AINovice2005/ModernBERT-base-lora-cicflow-1m-r4
AINovice2005/ModernBERT-base-lora-cicflow-1m-r16
cc: @davidberenstein1957
I previously remembered this done in the notus model release: argilla/notus-7b-v1
Examples:
AINovice2005/ModernBERT-base-lora-cicflow-1m-r8
AINovice2005/ModernBERT-base-lora-cicflow-1m-r4
AINovice2005/ModernBERT-base-lora-cicflow-1m-r16
cc: @davidberenstein1957
AINovice2005
posted an update 3 months ago
Post
479
Just published my first cuda kernel, inspired by Sage Attention. Feel free to try it out ☺️
AINovice2005/attention-int8
AINovice2005/attention-int8
- 2 replies