LEFT JOIN data jobs Alerts Saved
Jobs › GCMC Community Medical Center

Tech Lead Data Scientist, AI Evaluation & Monitoring

GCMC Community Medical Center · Work from Home
Apply on GCMC Community Medical Center's site
Tailor my resume
Posted
Apr 29, 2026
Last verified
1 h ago
Level
Senior
Function
Data
AI roleData sciencePythonGoSQLLLMsEvals

Open 6 months. Roles open this long are often filled, or kept open all year. Check the date on the employer's site before you apply.

Location:
Work from home (Pennsylvania)
Shift:
Days (United States of America)
Scheduled Weekly Hours:
40
Worker Type:
Regular
Exemption Status:
Yes
Job Summary:
The Tech Lead Data Scientist, AI Evaluation & Monitoring is the principal technical expert for how Geisinger evaluates, monitors, and optimizes AI systems in production. This is a hands-on technical leadership role. The Tech Lead sets the technical direction for AI evaluation across a large and growing portfolio, provides technical leadership to a team of data analysts who execute evaluation work, and partners directly with AI program teams to raise the quality of how AI is validated, monitored, and improved over time.

The role exists because AI at Geisinger has scaled past the point where oversight can be a document-review exercise. We need a technical leader who can guide program teams toward better-designed evaluations up front, instrument meaningful production monitoring, and continually advance the methods we use, from LLM-as-Judge frameworks to simulation-based testing to pragmatic experiment design that actually scales in healthcare.
Job Duties:
What You Will Own:

What You Will Not Own:

Shape of the Work:
This is a role that lives at three altitudes at once:

With program teams (hands-on advisory). Partner with program owners early, before evaluations are designed, to shape study approach, sample size, stratification, gold-standard definition, and decision thresholds. Translate ambiguous failure modes into concrete, defensible evaluation designs. Coach teams through the technical work so that what arrives at governance review is rigorous, not performative.
With the evaluation toolkit (hands-on build). Design and operate the reusable assets that let evaluation scale: LLM-as-Judge rubrics and calibration methods, golden sets, simulation harnesses, A/B and shadow-mode study templates, subgroup fairness analyses, and drift monitors. Keep a pragmatic eye on what actually works in a clinical environment versus what works in a paper.
With the analyst team (technical leadership). Set technical direction, assign work across active evaluations, review analysis code and study designs, and raise the technical bar. Mentor analysts on methodology, statistical rigor, and the domain knowledge that makes evaluation credible. Grow them from execution into independent evaluation design.

Methods You'll Use:

Work is typically performed in an office or remote environment. Accountable for satisfying all job specific obligations and complying with all organization policies and procedures. The specific statements in this profile are not intended to be all-inclusive. They represent typical elements considered necessary to successfully perform the job.

*Relevant experience may be a combination of related work experience and degree obtained (Master's Degree = 2 years; PHD = 4 years ).

Position Details:
Required Skills & Qualifications:

Education:
Bachelor's Degree-Related Field of Study (Required)
Experience:
Minimum of 6 years-Relevant experience* (Required)
Certification(s) and License(s):

Skills:
Analyzing, processing and building AI/ML solutions from Clinical and Operational data sources, such as Epic Clarity, HL7, DICOM, or ECG data, Clinical Databases, Communication, Critical Thinking, Data Analysis, Data Presentations, Group Collaboration, Leadership, Machine Learning Methods, Programming Languages, Structured Query Language (SQL)
OUR PURPOSE & VALUES: Everything we do is about caring for our patients, our members, our students, our Geisinger family and our communities.

We offer healthcare benefits for full time and part time positions from day one, including vision, dental and domestic partners. Perhaps just as important, we encourage an atmosphere of collaboration, cooperation and collegiality.

We know that a diverse workforce with unique experiences and backgrounds makes our team stronger. Our patients, members and community come from a wide variety of backgrounds, and it takes a diverse workforce to make better health easier for all. We are proud to be an affirmative action, equal opportunity employer and all qualified applicants will receive consideration for employment regardless to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or status as a protected veteran.

Apply on GCMC Community Medical Center's site

Left Join lists this role as published on GCMC Community Medical Center's own job board. Applications go to the employer.

Not this one? Get roles like it weekly

A job alert: new Data science roles from employers' own job boards, every Monday. Unsubscribe in one click.

Similar roles at other employers

All senior Data science roles →

More roles at GCMC Community Medical Center

All roles at GCMC Community Medical Center →