layiq
/la·yiq/adjective
worthy; deserving; fitting; suitable.
A role, opportunity, or path that merits attention, time, and pursuit.
Loading LAYIQ…
Job opportunity
[Proto] MTS - Software Engineer, Data Acqusition & Crawling
prometheus
San Francisco, California, United States
Source: prometheus careers · View original posting
Employer job description
From prometheus's posting. “We” and “our” refer to the employer.
The Role
The best models are trained on data nobody else bothered to collect. This role owns how we acquire the internet: a crawler and ingestion system that pulls text, code, images, video, audio, 3D assets, documents, and stranger formats from across the web at petabyte scale, and a standing search for sources other labs overlook.
You will design and run the distributed systems that do the fetching, extraction, dedup, and delivery, and you will spend real time hunting for new corpora about the physical world and the economy: engineering and manufacturing data, CAD and mesh repositories, sensor and telemetry streams, scientific archives, industrial and logistics records, public filings, maps and geospatial, and whatever else turns out to teach a model how the world actually works.
The job is half systems engineering at scale and half data research: every acquisition decision is measured by what it does to the model.
What you'll do
- Crawl & ingest: build and operate a large-scale, polite, fault-tolerant web crawler and the ingestion pipelines behind it, from scheduling and fetch to extraction, normalization, versioning, and delivery to the pretraining data team.