‍

Challenge Focus

The Policy2Code Prototyping Challenge seeks experiments to test the use of generative AI in the following scenarios:

• The translation of U.S. public benefits  policies into software code.
• The translation of U.S. public benefits policies into plain language logic models.
• The translation between coding languages, including converting code from legacy systems for benefits eligibility and enrollment. 
• Recommendations of additional scenarios that enhance policy implementation through digitization.
  

The demos and analysis from this challenge will be made available to the public, allowing practitioners to gain insights into the capabilities of current technologies and inspiring them to apply the latest advancements in their own projects through open-source code and available resources. We will be using Github Classroom to coordinate and share starter resources as well as instructions across the prototyping teams.

Benefits Policy Focus

The challenge is focused on U.S. public benefit programs like SNAP, WIC, Medicaid, TANF, Basic Income, Unemployment Insurance, SSI/SSDI, tax credits (EITC, CTC), and child care.  

‍

Data Use and Code Requirements‍

• All code should be well-documented. 
• All code and documentation used in the prototypes will be open and available for viewing by the public at the completion of the challenge. It may also be incorporated into Georgetown University research. 
• Only use policy documentation that is publicly available unless you have permission to share internal documentation.
• Ensure that you have the rights or permission to use the data or code in your prototype.  
• Ensure that no personally identifiable information (PII) is shared via your prototype or documentation. 
• Please only use a sandbox environment, not production systems.

‍

Who Is This For?

The challenge is open to individuals and teams including technologists, policy specialists, academics (including current or graduating students), program administrators, and other practitioners from all sectors who are interested in collaborating on the translation of U.S. public benefits policies into software code and logic models. Individuals are encouraged to join forces to form cross-disciplinary teams!

We believe that teams with at least one technologist and one policy specialist will benefit from collaboration between different domain experts. However, individuals with multi-dimensional expertise are welcome to participate. We encourage all teams to consider public sector and beneficiary perspectives as part of their prototype.

Please see the matchmaking section if you are looking for team members.

‍

How Does It Work?‍

1. Program Launch 

Join us for the Policy2Code launch webinar on May 3, 2024 from 1-2pm ET. We’ll provide a full overview of the program and leave time for questions. Questions and answers will be shared on this page following the webinar.

2. Apply

To participate in the Policy2Code Challenge, individuals or teams must complete an application form. The deadline for submission is May 22, 2024. The DBN and MDI will select up to 10 teams to join the challenge.

3. Announcement of Participants

We will announce selected teams the week of June 3, 2024. 

4. Monthly Touchpoints

Once selected, participants will have approximately three months to complete their testing.

Throughout the summer, we’ll host monthly touchpoints to share progress, solicit feedback from fellow participants, and receive support from coaches to help problem-solve issues with experiments. Teams are asked to join at least 2 of the 3 monthly touchpoints. 

Proposed dates: 

• June 25, 2024, 12 noon to 1:30pm ET
• July 23, 2024, 12 noon to 1:30pm ET
• August 22, 2024, 11:30am to 1pm ET

5. Demo Day at BenCon 2024 

The challenge will culminate in a Demo Day at the DBN’s signature event, BenCon 2024, scheduled for September 17-18 in Washington, D.C. Participants will present their experiments, prototypes, tinkering, and other developments to the community for feedback, awareness, and evaluation. 

Participating teams will be invited for in person demos, and a virtual option will also be available. Each member of a demo team will be awarded a $500 honorarium in accordance with the Georgetown University honoraria policy and if their institution allows. The coaches and organizers will also award non-monetary prizes for categories such as best use of open tools, best public sector demo, community choice, design simplicity, etc.

6. Report Findings 

In early 2025, the DBN and MDI will co-author and publish a report summarizing challenge activities, documenting findings from the prototypes, and highlighting the suitability of LLM frameworks and usage. Building on findings from the challenge, the public report will also recommend next steps for using cross-sector, multidisciplinary approaches to scaling and standardizing how benefits eligibility rules are written, tested, and implemented in digital systems for eligibility screening, program enrollment, and policy analysis.

‍

‍

Why Participate?

Understand the Application of Generative AI in Policy Implementation

Gain insights into how generative artificial intelligence technologies, specifically Large Language Models (LLMs), can be utilized to translate policy documents into plain language logic models and software code under the Rules as Code (RaC) approach.

Explore the Benefits and Limitations of Generative AI in Policy Translation

Analyze the potential benefits and limitations of using generative AI tools like LLMs for converting U.S. public benefits policies into software code, plain language logic models, and for translating between coding languages, with an emphasis on efficiency and accuracy.

Collaborate Across Disciplines for Policy Digitization

Discover the importance of cross-disciplinary collaboration between technologists, policy specialists, and practitioners across sectors in developing innovative solutions to translate policies into software code and logic models, applying the latest technological advancements.

Enhance Problem-solving Skills Through Experiments

Engage in hands-on experimentation to test scenarios related to policy digitization, such as converting legacy code systems for benefits eligibility, and recommend additional scenarios that can enhance policy implementation through digitization.

Apply RaC Approach in Real-world Applications

Gain practical experience in utilizing the Rules as Code approach to scale and standardize how benefits eligibility rules are written, tested, and implemented in digital systems, leading to a deeper understanding of digital service delivery.

‍

‍

Matchmaking

Are you looking for team members? 

Please use the matchmaking form and browse listings below.

If a listing does not have an email address, please email rulesascode@georgetown.edu and we will pass on your message.

Individuals Seeking Teams

Teams Seeking Team Members

‍

‍

Tools & Resources

We have collected tools and resources that may be useful in prototyping. Listing of a tool does not designate endorsement. If you’d like to recommend a tool or resource be added to the list, please send us a note at rulesascode@georgetown.edu

Digital Benefits Hub

The Digitizing Policy + Rules as Code page of the Digital Benefits Hub hosts numerous resources including example projects, case studies, demo videos, research, and more related to Rules as Code. 

The Automation + AI page of the Digital Benefits Hub hosts resources on how to responsibly, ethically, and equitably use artificial intelligence and automation, with strong consideration given to mitigating harms and bias.

Rules Frameworks

‍

LLMs + Tools 

Policy Manuals 

• SNAP policy manuals 

‍

Questions & Answers

Please see below for questions asked by potential participants and answers from the DBN and MDI Team. Please send any additional questions to rulesscode@georgetown.edu.

Question 1: I have a question regarding the source material. The website provides links to SNAP manuals and state websites. What should participants consider as the authoritative source for the rules to be used by the generative AI tools? Should it be from an official government website or a third-party source?

Answer 1: All of the mentioned sources are considered acceptable for the Policy2Code challenge. Many nonprofits and private sector organizations develop tools based on these rules, usually referencing original government documents.

There are some national policies that come from the U.S. Department of Agriculture,  the Food and Nutrition Service (FNS). For SNAP policies, the state policy manuals are considered the most reliable source. These manuals can be found in various formats, including dynamic websites and PDFs.  [To see examples of state rules, you can refer to the report authored by DBN-MDI: “Exploring Rules Communication: Moving Beyond Static Documents to Standardized Code for U.S. Public Benefits Programs”]

The SNAP policy manuals link listed on our Policy2Code website references a resource created by the Center on Budget and Policy Priorities. It provides comprehensive access to state SNAP websites and policy manuals. However, if you need information on programs like WIC in a specific state, a simple Google search will yield relevant results. It's worth noting that the quality and availability of the sources may vary.

Question 2: Will the coaches offer constructive criticism on the social potential for harm of these projects, or will they be focused primarily on technical assistance?

Answer 2: We have not yet publicly announced our coaches for the Policy2Code challenge, but we are actively working on assembling a team of multidisciplinary experts. These experts will go beyond specific domains such as technology, policy, AI, or AI ethics. We are considering cross-disciplinary support to ensure comprehensive guidance. At the DBN and MDI, we are deeply focused on understanding the civil rights implications of the tools being developed. We recognize that people's livelihoods and lives are significantly impacted by benefit systems, particularly with the introduction of AI and automation. Therefore, we are extremely cautious and strive to raise awareness about these implications.

Part of the purpose of the sandbox nature of this work is to showcase where these tools excel and where they may potentially cause harm. By experimenting away from real-life systems currently in use, we can better understand their performance and impact. Additionally, ethical considerations play a pivotal role in our approach. Comparing different approaches, such as open source versus closed source, can shed light on ethical considerations, biases, and better understanding of the tools. It is worth sharing these findings as part of your project's output.

Question 3: Could you elaborate on the output and deliverable and criteria for success?

Answer 3: We are really interested in trying to create a holistic setup to understand what works well and what doesn't when it comes to large language models and rules as code. There are various areas where your project can make an impact, and we don't expect you to figure out every aspect and provide a complete system. Instead, we encourage you to focus on a specific slice that interests you.

For example, if you're not as experienced in coding, you could explore the types of prompts that are effective for generating specific types of code using chatbots. On the other hand, if you excel in working with large language models, you might want to fine-tune them for SNAP rules or rules with specific properties. You could also investigate how to evaluate the performance of language models.

We have deliberately kept the task formulation broad, allowing you to choose a specific task that aligns with our overarching goal. The role of coaches and check-ins is to ensure that your chosen task fits the overall objective.

In summary, you have a task, you have an experiment, and you have results from that experiment. And then perhaps, you can explore some ethical considerations or policy considerations that are connected to your output.

Question 4: What are the expected actions for participants in this challenge? Are they expected to examine potential use cases and scenarios for applying AI and LLMs in information collection, decision-making, and the execution of public policy?

Answer 4: The challenge is focused on three core scenarios. The first scenario involves translating policy into software code and plain language, while the second scenario deals with translating code between different programming languages. The third scenario involves extracting code from legacy systems or transitioning from one code language to another for system integration purposes. While we are open to hearing ideas for other scenarios, our core research is focused on these three.

We are limiting the scope of the challenge to policy-focused scenarios specific to U.S. public benefit systems. While we are considering other programs, we are primarily concentrating on the U.S. context, as we are aware that compared to international colleagues, the Rules as Code concept is less advanced in the U.S. We hope to use this challenge to close that gap and produce new evidence and documentation on the potential benefits of implementing Rules as Code in the U.S.

Question 5: What is the criteria for selecting teams?

Answer 5: We do not plan to release a public-facing rubric. We have tried to keep the application form as low-burden as possible. And knowing that these are prototypes and experiments, and they're going to change over time. 

We would like to see teams with diverse expertise. We're very interested in teams that consider public sector or benefits perspectives, either  included within a prototyping team, or how you would engage those stakeholders directly. 

We've asked for a description of your intended prototype and how you plan to test for it. What technologies are you thinking about using? How are you mitigating bias and harms potentially in that system? How are you thinking about who is impacted by a potential system?

We're also wanting to make sure that we have a range of solutions and experiments. So we're also looking at diversity of experiments. So that we're able to learn more as a community. When developing your prototype, it is important not to scope it too large. Keep it within a specific scope to ensure you can complete the project successfully.

‍

About the Host Organizations

This challenge is organized by the Digital Benefits Network at the Beeck Center for Social Impact and Innovation in partnership with the Massive Data Institute, both based at Georgetown University in Washington, D.C.

The Digital Benefits Network (DBN) supports government in delivering public benefits services and technology that are accessible, effective, and equitable in order to ultimately increase economic opportunity. We explore current and near term challenges, while also creating space to envision future policies, services, and technologies. The DBN’s Rules as Code Community of Practice (RaC CoP) creates a shared learning and exchange space for people working on public benefits eligibility and enrollment systems — especially those tackling the issue of how policy becomes software code. The RaC CoP brings together cross-sector experts who share approaches, examples, and challenges.

At Georgetown’s McCourt School of Public Policy, the Massive Data Institute (MDI) is an interdisciplinary research institute that connects experts across computer science, data science, public health, public policy, and social science to tackle societal scale issues and impact public policy in ways that improves people’s lives through responsible evidence-based research.

Learn more about Rules as Code and join our community of practice on the Digital Benefits Hub. If you have further questions, email us at rulesascode@georgetown.edu.

‍

‍