ChartR Gyms Sign in
← Back to leaderboard
Claim #383348 · medium · COMMERCIAL

Claim #383348

BCBS OF NY EMPIRE PPO FS — work this professional (837P / CMS-1500) claim end-to-end and get it paid on exact-match coding.

Per-task leaderboard

Resolved out of 3 trials per model; expand a row to replay any trial.

1 Qwen3-32B (thinking) bedrock 2/3 resolved best 1.00
trial 1 Full resolution rwd 1.00 19 calls · 16d Replay →
trial 2 Full resolution rwd 0.80 31 calls · 32d Replay →
trial 3 Denied rwd 0.25 22 calls · 16d Replay →
2 GPT-5.5 openai 2/3 resolved best 0.80
trial 1 Full resolution rwd 0.80 27 calls · 32d Replay →
trial 2 Full resolution rwd 0.60 43 calls · 50d Replay →
trial 3 Paid (not fully resolved) rwd 0.34 34 calls · 48d Replay →
3 GPT-5.4 openai 1/3 resolved best 0.80
trial 1 Full resolution rwd 0.80 48 calls · 32d Replay →
trial 2 Paid (not fully resolved) rwd 0.29 63 calls · 48d Replay →
trial 3 Denied rwd 0.19 95 calls · 48d Replay →
4 Qwen3-235B (non-thinking) bedrock 0/3 resolved best 0.42
trial 1 Denied rwd 0.36 53 calls · 16d Replay →
trial 2 Denied rwd 0.31 58 calls · 16d Replay →
trial 3 Denied rwd 0.42 36 calls · 16d Replay →
5 Claude Opus 4.8 bedrock 0/3 resolved best 0.37
trial 1 Denied rwd 0.23 49 calls · 32d Replay →
trial 2 Denied rwd 0.37 72 calls · 16d Replay →
trial 3 Denied rwd 0.29 76 calls · 32d Replay →
Prompt
Environment
Verifier
Full task prompt · agent-facing, step 1
You are an expert hospital revenue-cycle specialist + medical coder working a professional (837P/CMS-1500) claim for payer 'BCBS OF NY EMPIRE PPO FS'.

You must fully resolve this professional (837P) claim. Steps:
  1. Verify active coverage (270/271) before billing.
  2. Coordinate benefits / set the correct primary payer when a secondary coverage exists.
  3. Review documentation; raise + resolve a provider query when the chart is ambiguous.
  4. Capture every billable charge from the chart (orders/MAR/supplies/time); add missed ones.
  5. Obtain a certified prior authorization for services that require it.
  6. Bill under the correct provider NPI + enrolled taxonomy for this payer.
  7. Translate SNOMED problems to ICD-10-CM at the right specificity, sequenced primary-first.
  8. Derive every CPT/HCPCS from the documented activity (the charge feed has NO codes), LEVEL the E/M from MDM/time, and set modifiers, units, POS, and dx pointers.
  9. For drug (J/Q) codes, report the 11-digit NDC + drug units (Box 24 shaded).
  10. Report the ordering/referring provider NPI (Box 17b) for ordered labs/imaging.
  11. Enter the certified prior-authorization number on the claim (Box 23).
  12. Pass the pre-submission scrub (NCCI/MUE/modifier/medical-necessity/NDC/referring).
  13. Submit the 837P and clear 999 + 277CA intake.
  14. Read the 835 ERA/EOB and post payment; reconcile patient responsibility.
  15. Work any denial: correct the claim and resubmit, or appeal with documentation.
  16. Survive any post-payment RAC/TPE audit by responding to the ADR on time.
A claim is PAID only if the coding matches exactly and all applicable steps are done; otherwise the payer denies it and you must rework or appeal it.

=== PATIENT CHART (read-only) ===
### patient_registration
Patient name: HERNANDEZ,SOPHIA
MRN: MRN000383348
Account #: ACCBCB7472F
Date of birth: 1947-03-15
Age: 77
Sex: F
Marital status: Divorced
Preferred language: Mandarin
Race: Declined
Ethnicity: Hispanic or Latino
Patient status: new (use 99202-99205 if new, 99212-99215 if established)

### contact
Home address: 3967 5TH AVE, NEW YORK, NY 10067
Phone: (212) 767-2867
Preferred pharmacy: WALGREENS #1187 - BROADWAY

### allergies
Drug allergies: No known drug allergies (NKDA)
Environmental: not on file
(reference only - not a billable item)

### insurance_card
Payer: BCBS OF NY EMPIRE PPO FS
Plan: COMMERCIAL
Member ID: MCCE92AA31
Group number: GRP-BAEE
Subscriber: HERNANDEZ,SOPHIA (self)
Relationship to subscriber: self

### care_team
Primary care provider (PCP): RODRIGUEZ,LUIS DO
  NPI: 1962435981
Rendering provider (this visit): PATEL,RAJ MD
  NPI: 1700802145
Service facility: CHARTR HEALTH - DOWNTOWN CLINIC

### provider_directory
Rendering provider: PATEL,RAJ MD
  NPI: 1700802145
Billing provider: CHARTR HEALTH GROUP
  NPI: 1064830069
  Tax ID (EIN): 97-7832867

### progress_note
### [visit]
Date:   |  Start Time: 11:23:00  |  End Time: 11:38:00  |  Length of Stay: 15 min  |  Reason: Throat hurts bad hard time swollowing  |  Visit type: UC - NEW
Provider: 

Status: CHK

### [clinical_note_(free_text)]
HPI: sore throat with difficulty swallowing, fever,. chills, , Onset:  gradual, Severity:  mild
[ORAL CAVITY:] mucosa moist, palate normal
[LUNGS:] clear to auscultation bilaterally, no wheezes, rales, rhonchi
[HEART:] no murmurs, regular rate and rhythm, S1, S2 normal
[NECK/THYROID:] neck supple, full range of motion, no cervical lymphadenopathy
[THROAT:] Erythematous posterior pharynx. Large white lesions on tonnsils
[GENERAL APPEARANCE:] in no acute distress, well developed, well nourished
Billing: Haines,Jax  04:27:46 PM > No changes.
Kendal,Blumenthal   6:44:51 PM > Checked with Epaces, found that member is active on DOS with effective date of , (Urgent Care) Co-pay = $0, Co-insurance = 0%, Ded./Left=$0/$0(as per portal)
Family: unremarkable
PrevMed: Drink plenty of fluids and stay hydrated. 
Get plenty of rest. 
If symptoms worsen then go to the ER for further evaluation. 
Follow up with PCP in 2-3 days
Specimens taken today are sent to the lab for further testing, if they come back positive we will notify you and prescribe additional medication if needed.
Social: unremarkable

### [vitals]
Temperature:  ( 11:40:10)
Blood Pressure:  ( 11:40:10)
Heart Rate:  ( 11:40:10)
Respiratory Rate:  ( 11:40:10)
Height:  ( 11:40:10)
Weight:  ( 11:40:10)
BMI:  ( 11:40:10)
Oxygen Saturation %:  ( 11:40:10)
Height (cm):  ( 11:40:10)
Weight (kg):  ( 11:40:10)

### [diagnoses]
SNOMED 43878008 (primary)

### [disposition]
N/A

### [assessment_and_plan]
N/A

--- ORDERS (from progress-note XML) ---
N/A

--- DOCUMENTS (attached to encounter) ---
Document count: 1
  docID 7506272  |  type: Speciality Forms  |  name: Tobacco Control  11:40:51  |  description: EMR Form  |  scanDate:  11:40:51

### [orders_(from_progress-note_xml)]
N/A

### problem_list_snomed
Diagnoses are recorded in SNOMED CT and are NOT auto-translated - read each documented condition and assign the specific ICD-10-CM yourself (choose the right laterality/acuity/specificity from the note, then verify the code with kb_lookup_icd10 code=...):
  - SNOMED 43878008 (primary): see note

### visit_meta
Date of service: 2024-09-15
Facility: CHARTR HEALTH - DOWNTOWN CLINIC
Place of service: 11 (Office)
Rendering provider: PATEL,RAJ MD
Visit type: UC - NEW
New patient: yes
Visit minutes: not documented
Problems addressed: 1

### ordering_provider
Rendering / ordering provider: PATEL,RAJ MD (NPI 1700802145). For ordered services (imaging/diagnostics) report this NPI as the ordering/referring provider on the claim (Box 17b).

=== AVAILABLE TOOLS ===
  - read_chart({"section": "progress_note"}): Read EHR chart sections (optionally one section).
  - get_status({}): Get the current case status / stage.
  - kb_lookup_icd10({"code": "J02.0"}): Verify an ICD-10-CM code's official description (code -> description). No clinical-term search and no SNOMED auto-translate - YOU assign the code from the documentation, then confirm it here.
  - kb_lookup_cpt({"code": "87880"}): Verify a CPT/HCPCS code's description (code -> description; drug J/Q codes also return their NDC). No clinical-term search - YOU derive the code from the documented activity, then confirm it here.
  - kb_coding_guidelines({"topic": "icd10"}): Reference coding rules (topic=icd10|em|service_lines|modifiers|medical_necessity|workflow, or omit for all). Rules only - you assign the codes from the chart.
  - kb_lookup_modifier({"modifier": "25"}): Look up a CPT modifier's meaning.
  - kb_lookup_carc({"carc": "CO-197"}): Look up a CARC denial code + remedy.
  - kb_lookup_ncci({"cpt_a": "99213", "cpt_b": "20610"}): Check an NCCI procedure-to-procedure pair.
  - kb_lookup_payer_policy({}): Look up this payer's companion-guide policy.
  - kb_lookup_em_guidelines({"level": "99214"}): Look up E/M level guidelines.
  - portal_login({}): Start payer-portal login (triggers OTP).
  - portal_verify_otp({"code": "000000"}): Verify the portal OTP code.
  - portal_member_search({}): Search the payer portal for the member.
  - portal_download_era({}): Download the latest ERA/EOB (835).
  - check_eligibility({}): Run a 270/271 eligibility check.
  - coordinate_benefits({"primary_payer": "MEDICARE"}): Set the primary payer for COB.
  - review_documentation({}): Coder review of the chart documentation.
  - create_provider_query({"topic": "laterality", "question": "Which knee?"}): Send a clarification query to the provider.
  - resolve_provider_query({"query_id": "q-..."}): Read the provider's query response.
  - capture_charges({}): Aggregate billable charges from the chart.
  - add_charge({"code": "87880"}): Manually add a missed charge.
  - submit_prior_auth({"service_codes": ["70553"]}): Submit a 278 prior-auth request.
  - attach_auth_docs({}): Attach clinical docs to the auth.
  - check_prior_auth({}): Poll the prior-auth status.
  - update_provider_identity({"billing_taxonomy": "261QU0200X"}): Correct billing NPI/taxonomy/TIN.
  - set_diagnoses({"codes": ["J02.0", "R50.9"]}): Set Box 21 ICD-10-CM diagnoses (primary first).
  - set_service_lines({"lines": [{"cpt": "99213", "modifiers": ["25"], "units": 1, "dx_pointers": [1], "place_of_service": "20"}]}): Set Box 24 CPT/HCPCS service lines (incl. NDC for drug codes).
  - set_claim_header({"referring_npi": "1999999984", "prior_auth_number": "__certified__"}): Set CMS-1500 header boxes (14 onset, 17/17b referring, 23 prior-auth #, 10 accident, 19 notes).
  - run_scrub({}): Run the pre-submission claim scrub (NCCI/MUE/NDC/referring/etc.).
  - submit_claim({}): Build + submit the 837P (runs 999/277CA/835).
  - post_remittance({}): Post a paid ERA and reconcile.
  - attach_documentation({"document_id": "DOC-..."}): Attach a clinical document (PWK) for appeal/audit.
  - file_appeal({"track": "narrative", "rationale": "..."}): File an appeal (corrected_claim or narrative). Only allowed after all 3 submission attempts are exhausted.
  - respond_to_audit({}): Submit records in response to an audit ADR.
  - rebut_audit({}): Rebut an audit recoupment.
  - write_off({}): Write off the balance and close the case.
  - done({}): Signal the case is fully worked.

Lifecycle stages
intakeeligibilitydocumentationcharge_captureauthorizationcodingidentityscrubsubmittedremittancedenialappealauditclosed_paidclosed_writeoff
Tool surface · 37
read_chart · get_status · kb_lookup_icd10 · kb_lookup_cpt · kb_coding_guidelines · kb_lookup_modifier · kb_lookup_carc · kb_lookup_ncci · kb_lookup_payer_policy · kb_lookup_em_guidelines · portal_login · portal_verify_otp · portal_member_search · portal_download_era · check_eligibility · coordinate_benefits · review_documentation · create_provider_query · resolve_provider_query · capture_charges · add_charge · submit_prior_auth · attach_auth_docs · check_prior_auth · update_provider_identity · set_diagnoses · set_service_lines · set_claim_header · run_scrub · submit_claim · post_remittance · attach_documentation · file_appeal · respond_to_audit · rebut_audit · write_off · done
Paid only on exact-match coding The claim is graded against hidden gold (real historical adjudication). Terminal, episode-level reward - not per tool call.
Reward formula
reward = clamp(0.50·outcome + 0.35·coding + 0.15·process, 0, 1) × (0.90 + 0.10·efficiency) × first_pass_factor
Hidden gold · answer key
CPT / HCPCS
87880, 99203
ICD-10-CM
J02.0
E/M level
99203